vdusek commented on pull request #6722: URL: https://github.com/apache/airflow/pull/6722#issuecomment-877089233
@ashb We're having the same troubles after upgrade to 2.1.X. I'd say around 5% of our tasks are failing with the following: Log on webserver: ``` *** Falling back to local log *** Log file does not exist: /home/airflow/logs/dm-bi-edw-c360-export-brg/ext_w_brg_bi_c360_t_customer_support/2021-07-08T10:00:00+00:00/1.log *** Fetching from: http://:8793/log/dm-bi-edw-c360-export-brg/ext_w_brg_bi_c360_t_customer_support/2021-07-08T10:00:00+00:00/1.log *** Failed to fetch log file from worker. The request to ':///' is missing either an 'http://' or 'https://' protocol. ``` Log on flower: ``` Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__ return self.run(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 88, in execute_command _execute_in_fork(command_to_exec) File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 99, in _execute_in_fork raise AirflowException('Celery command failed on host: ' + get_hostname()) airflow.exceptions.AirflowException: Celery command failed on host: 172.20.38.183 ``` Log on scheduler: ``` [2021-07-09 10:00:59,824] {{scheduler_job.py:1258}} ERROR - Executor reports task instance <TaskInstance: dm-bi-edw-c360-export-brg.ext_w_brg_bi_c360_t_customer_support 2021-07-08 10:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally? [2021-07-09 10:03:59,831] {{timeout.py:36}} ERROR - Process timed out, PID: 15223 [2021-07-09 10:09:11,547] {{dagrun.py:429}} ERROR - Marking run <DagRun dm-bi-edw-c360-export-brg @ 2021-07-08 10:00:00+00:00: scheduled__2021-07-08T10:00:00+00:00, externally triggered: False> failed ``` But there's no log on the worker. It seems to me like the task was never executed on the worker at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
