I get the same error in airflow-worker.err, the real underlying cause only
revealed itself in airflow-worker.out
Damian
-----Original Message-----
From: heng gu <heng...@yahoo.com.INVALID>
Sent: Wednesday, February 26, 2020 12:35
To: dev@airflow.apache.org
Subject: Re: task failed without running
thanks for your help! I have similar DB timeout problem with other tasks, but
not this one. The error message I got for this one from airflow-worker.err is
File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67,
in execute_command close_fds=True, env=env) File
"pypacks/lib/python3.6/subprocess.py", line 311, in check_call raise
CalledProcessError(retcode, cmd) Apparently Celery executor ran the worker in
subprocess.py and returned non-zero retcode.
Heng
On Wednesday, February 26, 2020, 09:57:29 AM EST, Shaw, Damian P.
<damian.sha...@credit-suisse.com> wrote:
FYI,
I've had this issue for a few different reasons, the first was that my worker
failed to read the file (this for me was a permission issue). Now I'm having it
a second time and it appears Airflow timeouts connecting to the DB while
setting up the task, for me I can see this in the stdout of the Airflow worker
(not in stderr though).
I am looking to increase the connect timeout using this config item:
https://airflow.readthedocs.io/en/latest/configurations-ref.html#sql-alchemy-connect-args
and this sqlalchemy option: https://stackoverflow.com/a/35640876/2958068
(side note: concerningly this option doesn't seem to be documented in the new
documentation https://airflow.apache.org/docs/stable/configurations-ref.html .
But I found the exact code in question:
https://github.com/apache/airflow/blob/6ec9664a6be151e8fa94ef207cd98b459d392af2/airflow/settings.py#L187
)
This is a snippet of the exception I am getting, it seems to be caused during
session.merge(self) in _check_and_change_state_before_execution. Check if you
are logging your Airflow worker stdout somewhere and see if you have a similar
exception:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 583, in
connect
**kwargs)
File ".../lib/python3.7/socket.py", line 727, in create_connection
raise err
File ".../lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".../lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2276,
in _wrap_pool_connect
return fn()
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in
connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on
'my_db.my_domain.net' (timed out)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File ".../bin/airflow", line 37, in <module>
args.func(args)
...
File ".../lib/python3.7/site-packages/airflow/models/taskinstance.py", line
852, in _check_and_change_state_before_execution
session.merge(self)
...
File ".../lib/python3.7/site-packages/pymysql/connections.py", line 630, in
connect
raise exc
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't
connect to MySQL server on 'my_db.my_domain.net' (timed out)")
Regards
Damian
-----Original Message-----
From: Mehmet Ersoy <mehmet.ersoy1...@gmail.com>
Sent: Wednesday, February 26, 2020 03:18
To: dev@airflow.apache.org
Subject: Re: task failed without running
Hi Heng,
I have same problem. Failed jobs do not dump log files. Do you have this
problem too?
Have you been using Redis as message queue? Also can you send configurations
related parallelism and concurrent in airflow.cfg file?
Best regards,
Mehmet.
heng gu <heng...@yahoo.com.invalid>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu
yazdı:
> I have this dag with a branchpythonoperator task kicking off many of
> 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the
> other two (register_YZ, register_ZY) were failed without running (see
> the attached UI screen shots). There is no log for tasks register_YZ
> and register_ZY. I am using Celery Executor and running 12 workers
> executing register_XX tasks. I am using airflow version 1.10.6. Any idea how
> to fix it?
>
--
Mehmet ERSOY
===============================================================================
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
===============================================================================
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
