raphaelsimeon commented on issue #33688: URL: https://github.com/apache/airflow/issues/33688#issuecomment-1697700661
Hello there ! Following [this Slack discussion,](https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1693298060235109) I'm adding a few logs for a similar issue : - Upgrading from 2.5.1 to 2.7.0 - **Some** tasks are taking longer than usual (see partial logs here where time between each other increased significantly) Before upgrade: `[2023-08-16, 18:10:39 CEST] {{standard_task_runner.py:83}} INFO - Job 966179: Subtask upload_cards_file_to_s3 [2023-08-16, 18:10:39 CEST] {{task_command.py:388}} INFO - Running <TaskInstance: get_chargebee_exports.upload_cards_file_to_s3 scheduled__2023-08-16T15:10:00+00:00 [running]> on host ip-10-0-0-157.eu-west-1.compute.internal [2023-08-16, 18:10:39 CEST] {{taskinstance.py:1507}} INFO - Exporting the following env vars:` Executed within a second After upgrade: `[2023-08-29, 16:12:21 CEST] {{standard_task_runner.py:85}} INFO - Job 985071: Subtask upload_cards_file_to_s3 [2023-08-29, 16:12:28 CEST] {{task_command.py:415}} INFO - Running <TaskInstance: get_chargebee_exports.upload_cards_file_to_s3 scheduled__2023-08-29T13:10:00+00:00 [running]> on host ip-10-0-0-157.eu-west-1.compute.internal [2023-08-29, 16:12:34 CEST] {{taskinstance.py:1660}} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='get_chargebee_exports' AIRFLOW_CTX_TASK_ID='upload_cards_file_to_s3' AIRFLOW_CTX_EXECUTION_DATE='2023-08-29T13:10:00+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='scheduled__2023-08-29T13:10:00+00:00' ` Taking 13 seconds The rest of the task takes the same amount of time to execute A few things I noticed : - This happens only in my production setup, not in my dev and preprod environments, which are very similar in terms of setup. Only a few things are different, namely: * Production setup runs on a bigger AWS instance than preprod and dev (local environment so personal laptops) * Most of my dags are turned ON on production, most of dags are OFF in other environments, could it be that the scheduler is impacted by this ? * Metadata db runs on RDS for prod and preprod with same instance type, for dev it's a local postgres db. - This happens only when several dags are running at the same time, or with dags that contain several task to run in parallel/schedule, meaning a dag that's just task A then task B that runs in a calm period of the day does not look impacted (Can edit my post with more details if necessary) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
