FYI, I've had this issue for a few different reasons, the first was that my worker failed to read the file (this for me was a permission issue). Now I'm having it a second time and it appears Airflow timeouts connecting to the DB while setting up the task, for me I can see this in the stdout of the Airflow worker (not in stderr though). I am looking to increase the connect timeout using this config item: https://airflow.readthedocs.io/en/latest/configurations-ref.html#sql-alchemy-connect-args and this sqlalchemy option: https://stackoverflow.com/a/35640876/2958068
(side note: concerningly this option doesn't seem to be documented in the new documentation https://airflow.apache.org/docs/stable/configurations-ref.html . But I found the exact code in question: https://github.com/apache/airflow/blob/6ec9664a6be151e8fa94ef207cd98b459d392af2/airflow/settings.py#L187 ) This is a snippet of the exception I am getting, it seems to be caused during session.merge(self) in _check_and_change_state_before_execution. Check if you are logging your Airflow worker stdout somewhere and see if you have a similar exception: Traceback (most recent call last): File ".../lib/python3.7/site-packages/pymysql/connections.py", line 583, in connect **kwargs) File ".../lib/python3.7/socket.py", line 727, in create_connection raise err File ".../lib/python3.7/socket.py", line 716, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File ".../lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect return fn() ... File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect raise exc pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)") The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".../bin/airflow", line 37, in <module> args.func(args) ... File ".../lib/python3.7/site-packages/airflow/models/taskinstance.py", line 852, in _check_and_change_state_before_execution session.merge(self) ... File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in connect raise exc sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'my_db.my_domain.net' (timed out)") Regards Damian -----Original Message----- From: Mehmet Ersoy <mehmet.ersoy1...@gmail.com> Sent: Wednesday, February 26, 2020 03:18 To: dev@airflow.apache.org Subject: Re: task failed without running Hi Heng, I have same problem. Failed jobs do not dump log files. Do you have this problem too? Have you been using Redis as message queue? Also can you send configurations related parallelism and concurrent in airflow.cfg file? Best regards, Mehmet. heng gu <heng...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu yazdı: > I have this dag with a branchpythonoperator task kicking off many of > 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the > other two (register_YZ, register_ZY) were failed without running (see > the attached UI screen shots). There is no log for tasks register_YZ > and register_ZY. I am using Celery Executor and running 12 workers > executing register_XX tasks. I am using airflow version 1.10.6. Any idea how > to fix it? > -- Mehmet ERSOY =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
