raphaelsimeon commented on issue #33688:
URL: https://github.com/apache/airflow/issues/33688#issuecomment-1697700661

   Hello there !
   Following [this Slack 
discussion,](https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1693298060235109)
 I'm adding a few logs for a similar issue :
   - Upgrading from 2.5.1 to 2.7.0
   - **Some** tasks are taking longer than usual (see partial logs here where 
time between each other increased significantly)
   Before upgrade:
   `[2023-08-16, 18:10:39 CEST] {{standard_task_runner.py:83}} INFO - Job 
966179: Subtask upload_cards_file_to_s3
   [2023-08-16, 18:10:39 CEST] {{task_command.py:388}} INFO - Running 
<TaskInstance: get_chargebee_exports.upload_cards_file_to_s3 
scheduled__2023-08-16T15:10:00+00:00 [running]> on host 
ip-10-0-0-157.eu-west-1.compute.internal
   [2023-08-16, 18:10:39 CEST] {{taskinstance.py:1507}} INFO - Exporting the 
following env vars:`
   Executed within a second
   
   After upgrade:
   `[2023-08-29, 16:12:21 CEST] {{standard_task_runner.py:85}} INFO - Job 
985071: Subtask upload_cards_file_to_s3
   [2023-08-29, 16:12:28 CEST] {{task_command.py:415}} INFO - Running 
<TaskInstance: get_chargebee_exports.upload_cards_file_to_s3 
scheduled__2023-08-29T13:10:00+00:00 [running]> on host 
ip-10-0-0-157.eu-west-1.compute.internal
   [2023-08-29, 16:12:34 CEST] {{taskinstance.py:1660}} INFO - Exporting env 
vars: AIRFLOW_CTX_DAG_OWNER='airflow' 
AIRFLOW_CTX_DAG_ID='get_chargebee_exports' 
AIRFLOW_CTX_TASK_ID='upload_cards_file_to_s3' 
AIRFLOW_CTX_EXECUTION_DATE='2023-08-29T13:10:00+00:00' 
AIRFLOW_CTX_TRY_NUMBER='1' 
AIRFLOW_CTX_DAG_RUN_ID='scheduled__2023-08-29T13:10:00+00:00'
   `
   Taking 13 seconds
   The rest of the task takes the same amount of time to execute
   
   A few things I noticed : 
   - This happens only in my production setup, not in my dev and preprod 
environments, which are very similar in terms of setup. Only a few things are 
different, namely: 
       * Production setup runs on a bigger AWS instance than preprod and dev 
(local environment so personal laptops)
       * Most of my dags are turned ON on production, most of dags are OFF in 
other environments, could it be that the scheduler is impacted by this ?
       * Metadata db runs on RDS for prod and preprod with same instance type, 
for dev it's a local postgres db.
   - This happens only when several dags are running at the same time, or with 
dags that contain several task to run in parallel/schedule, meaning a dag 
that's just task A then task B that runs in a calm period of the day does not 
look impacted
   
   (Can edit my post with more details if necessary)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to