[ https://issues.apache.org/jira/browse/AIRFLOW-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kengo Seki reassigned AIRFLOW-1235: ----------------------------------- Assignee: Kengo Seki > Odd behaviour when all gunicorn workers die > ------------------------------------------- > > Key: AIRFLOW-1235 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1235 > Project: Apache Airflow > Issue Type: Bug > Components: webserver > Affects Versions: 1.8.0 > Reporter: Erik Forsberg > Assignee: Kengo Seki > > The webserver has sometimes stopped responding to port 443, and today I found > the issue - I had a misconfigured resolv.conf that made it unable to talk to > my postgresql. This was the root cause, but the way airflow webserver behaved > was a bit odd. > It seems that when all gunicorn workers failed to start, the gunicorn master > shut down. However, the main process (the one that starts gunicorn master) > did not shut down, so there was no way of detecting the failed status of > webserver from e.g. systemd or init script. > Full traceback leading to stale webserver process: > {noformat} > May 21 09:51:57 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] > [23794] [ERROR] Exception in worker process: > May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last): > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1122, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, > self._timeout) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py", > line 145, in get > May 21 09:51:57 airmaster01 airflow[26451]: raise Empty > May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty > May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above > exception, another exception occurred: > May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last): > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2147, in _wrap_pool_connect > May 21 09:51:57 airmaster01 airflow[26451]: return fn() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 387, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return > _ConnectionFairy._checkout(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 766, in _checkout > May 21 09:51:57 airmaster01 airflow[26451]: fairy = > _ConnectionRecord.checkout(pool) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 516, in checkout > May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1138, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", > line 66, in __exit__ > May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, > exc_value, exc_tb) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 187, in reraise > May 21 09:51:57 airmaster01 airflow[26451]: raise value > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1135, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 333, in _create_connection > May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 461, in __init__ > May 21 09:51:57 airmaster01 airflow[26451]: > self.__connect(first_connect_check=True) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 651, in __connect > May 21 09:51:57 airmaster01 airflow[26451]: connection = > pool._invoke_creator(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", > line 105, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, > **cparams) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", > line 393, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, > **cparams) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", > line 130, in connect > May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, > connection_factory=connection_factory, **kwasync) > May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could > not translate host name "airstorage01.example.com" to address: Name or > service not known > May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the > direct cause of the following exception: > May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last): > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", > line 507, in spawn_worker > May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py", > line 118, in init_process > May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py", > line 67, in wsgi > May 21 09:51:57 airmaster01 airflow[26451]: self.callable = self.load() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", > line 65, in load > May 21 09:51:57 airmaster01 airflow[26451]: return self.load_wsgiapp() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", > line 52, in load_wsgiapp > May 21 09:51:57 airmaster01 airflow[26451]: return > util.import_app(self.app_uri) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line > 366, in import_app > May 21 09:51:57 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__) > May 21 09:51:57 airmaster01 airflow[26451]: File "<string>", line 1, in > <module> > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", > line 161, in cached_app > May 21 09:51:57 airmaster01 airflow[26451]: app = create_app(config) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", > line 60, in create_app > May 21 09:51:57 airmaster01 airflow[26451]: from airflow.www import views > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", > line 1975, in <module> > May 21 09:51:57 airmaster01 airflow[26451]: class > ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView): > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", > line 2057, in ChartModelView > May 21 09:51:57 airmaster01 airflow[26451]: > .group_by(models.Connection.conn_id) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2855, in __iter__ > May 21 09:51:57 airmaster01 airflow[26451]: return > self._execute_and_instances(context) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2876, in _execute_and_instances > May 21 09:51:57 airmaster01 airflow[26451]: close_with_result=True) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2885, in _get_bind_args > May 21 09:51:57 airmaster01 airflow[26451]: **kw > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2867, in _connection_from_session > May 21 09:51:57 airmaster01 airflow[26451]: conn = > self.session.connection(**kw) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", > line 966, in connection > May 21 09:51:57 airmaster01 airflow[26451]: cannot otherwise be identified. > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", > line 971, in _connection_for_bind > May 21 09:51:57 airmaster01 airflow[26451]: closed. This flag only has an > effect if this :class:`.Session` is > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", > line 403, in _connection_for_bind > May 21 09:51:57 airmaster01 airflow[26451]: conn = bind.contextual_connect() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2112, in contextual_connect > May 21 09:51:57 airmaster01 airflow[26451]: > self._wrap_pool_connect(self.pool.connect, None), > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2151, in _wrap_pool_connect > May 21 09:51:57 airmaster01 airflow[26451]: e, dialect, self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 1465, in _handle_dbapi_exception_noconnection > May 21 09:51:57 airmaster01 airflow[26451]: exc_info > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 203, in raise_from_cause > May 21 09:51:57 airmaster01 airflow[26451]: reraise(type(exception), > exception, tb=exc_tb, cause=cause) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 186, in reraise > May 21 09:51:57 airmaster01 airflow[26451]: raise value.with_traceback(tb) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2147, in _wrap_pool_connect > May 21 09:51:57 airmaster01 airflow[26451]: return fn() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 387, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return > _ConnectionFairy._checkout(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 766, in _checkout > May 21 09:51:57 airmaster01 airflow[26451]: fairy = > _ConnectionRecord.checkout(pool) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 516, in checkout > May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1138, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", > line 66, in __exit__ > May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, > exc_value, exc_tb) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 187, in reraise > May 21 09:51:57 airmaster01 airflow[26451]: raise value > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1135, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 333, in _create_connection > May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 461, in __init__ > May 21 09:51:57 airmaster01 airflow[26451]: > self.__connect(first_connect_check=True) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 651, in __connect > May 21 09:51:57 airmaster01 airflow[26451]: connection = > pool._invoke_creator(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", > line 105, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, > **cparams) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", > line 393, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, > **cparams) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", > line 130, in connect > May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, > connection_factory=connection_factory, **kwasync) > May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: > (psycopg2.OperationalError) could not translate host name > "airstorage01.example.com" to address: Name or service not known > May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last): > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1122, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: return self._pool.get(wait, > self._timeout) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/queue.py", > line 145, in get > May 21 09:51:57 airmaster01 airflow[26451]: raise Empty > May 21 09:51:57 airmaster01 airflow[26451]: sqlalchemy.util.queue.Empty > May 21 09:51:57 airmaster01 airflow[26451]: During handling of the above > exception, another exception occurred: > May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last): > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2147, in _wrap_pool_connect > May 21 09:51:57 airmaster01 airflow[26451]: return fn() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 387, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return > _ConnectionFairy._checkout(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 766, in _checkout > May 21 09:51:57 airmaster01 airflow[26451]: fairy = > _ConnectionRecord.checkout(pool) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 516, in checkout > May 21 09:51:57 airmaster01 airflow[26451]: rec = pool._do_get() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1138, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: self._dec_overflow() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", > line 66, in __exit__ > May 21 09:51:57 airmaster01 airflow[26451]: compat.reraise(exc_type, > exc_value, exc_tb) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 187, in reraise > May 21 09:51:57 airmaster01 airflow[26451]: raise value > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1135, in _do_get > May 21 09:51:57 airmaster01 airflow[26451]: return self._create_connection() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 333, in _create_connection > May 21 09:51:57 airmaster01 airflow[26451]: return _ConnectionRecord(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 461, in __init__ > May 21 09:51:57 airmaster01 airflow[26451]: > self.__connect(first_connect_check=True) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 651, in __connect > May 21 09:51:57 airmaster01 airflow[26451]: connection = > pool._invoke_creator(self) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", > line 105, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return dialect.connect(*cargs, > **cparams) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", > line 393, in connect > May 21 09:51:57 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, > **cparams) > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", > line 130, in connect > May 21 09:51:57 airmaster01 airflow[26451]: conn = _connect(dsn, > connection_factory=connection_factory, **kwasync) > May 21 09:51:57 airmaster01 airflow[26451]: psycopg2.OperationalError: could > not translate host name "airstorage01.example.com" to address: Name or > service not known > May 21 09:51:57 airmaster01 airflow[26451]: The above exception was the > direct cause of the following exception: > May 21 09:51:57 airmaster01 airflow[26451]: Traceback (most recent call last): > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/arbiter.py", > line 507, in spawn_worker > May 21 09:51:57 airmaster01 airflow[26451]: worker.init_process() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/workers/base.py", > line 118, in init_process > May 21 09:51:57 airmaster01 airflow[26451]: self.wsgi = self.app.wsgi() > May 21 09:51:57 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/base.py", > line 67, in wsgi > May 21 09:51:58 airmaster01 airflow[26451]: self.callable = self.load() > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", > line 65, in load > May 21 09:51:58 airmaster01 airflow[26451]: return self.load_wsgiapp() > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/app/wsgiapp.py", > line 52, in load_wsgiapp > May 21 09:51:58 airmaster01 airflow[26451]: return > util.import_app(self.app_uri) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/gunicorn/util.py", line > 366, in import_app > May 21 09:51:58 airmaster01 airflow[26451]: app = eval(obj, mod.__dict__) > May 21 09:51:58 airmaster01 airflow[26451]: File "<string>", line 1, in > <module> > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", > line 161, in cached_app > May 21 09:51:58 airmaster01 airflow[26451]: app = create_app(config) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/app.py", > line 60, in create_app > May 21 09:51:58 airmaster01 airflow[26451]: from airflow.www import views > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", > line 1975, in <module> > May 21 09:51:58 airmaster01 airflow[26451]: class > ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView): > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/airflow/www/views.py", > line 2057, in ChartModelView > May 21 09:51:58 airmaster01 airflow[26451]: > .group_by(models.Connection.conn_id) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2855, in __iter__ > May 21 09:51:58 airmaster01 airflow[26451]: return > self._execute_and_instances(context) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2876, in _execute_and_instances > May 21 09:51:58 airmaster01 airflow[26451]: close_with_result=True) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2885, in _get_bind_args > May 21 09:51:58 airmaster01 airflow[26451]: **kw > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/query.py", > line 2867, in _connection_from_session > May 21 09:51:58 airmaster01 airflow[26451]: conn = > self.session.connection(**kw) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", > line 966, in connection > May 21 09:51:58 airmaster01 airflow[26451]: cannot otherwise be identified. > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", > line 971, in _connection_for_bind > May 21 09:51:58 airmaster01 airflow[26451]: closed. This flag only has an > effect if this :class:`.Session` is > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/orm/session.py", > line 403, in _connection_for_bind > May 21 09:51:58 airmaster01 airflow[26451]: conn = bind.contextual_connect() > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2112, in contextual_connect > May 21 09:51:58 airmaster01 airflow[26451]: > self._wrap_pool_connect(self.pool.connect, None), > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2151, in _wrap_pool_connect > May 21 09:51:58 airmaster01 airflow[26451]: e, dialect, self) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 1465, in _handle_dbapi_exception_noconnection > May 21 09:51:58 airmaster01 airflow[26451]: exc_info > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 203, in raise_from_cause > May 21 09:51:58 airmaster01 airflow[26451]: reraise(type(exception), > exception, tb=exc_tb, cause=cause) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 186, in reraise > May 21 09:51:58 airmaster01 airflow[26451]: raise value.with_traceback(tb) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/base.py", > line 2147, in _wrap_pool_connect > May 21 09:51:58 airmaster01 airflow[26451]: return fn() > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 387, in connect > May 21 09:51:58 airmaster01 airflow[26451]: return > _ConnectionFairy._checkout(self) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 766, in _checkout > May 21 09:51:58 airmaster01 airflow[26451]: fairy = > _ConnectionRecord.checkout(pool) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 516, in checkout > May 21 09:51:58 airmaster01 airflow[26451]: rec = pool._do_get() > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1138, in _do_get > May 21 09:51:58 airmaster01 airflow[26451]: self._dec_overflow() > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", > line 66, in __exit__ > May 21 09:51:58 airmaster01 airflow[26451]: compat.reraise(exc_type, > exc_value, exc_tb) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/util/compat.py", > line 187, in reraise > May 21 09:51:58 airmaster01 airflow[26451]: raise value > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 1135, in _do_get > May 21 09:51:58 airmaster01 airflow[26451]: return self._create_connection() > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 333, in _create_connection > May 21 09:51:58 airmaster01 airflow[26451]: return _ConnectionRecord(self) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 461, in __init__ > May 21 09:51:58 airmaster01 airflow[26451]: > self.__connect(first_connect_check=True) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/pool.py", > line 651, in __connect > May 21 09:51:58 airmaster01 airflow[26451]: connection = > pool._invoke_creator(self) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", > line 105, in connect > May 21 09:51:58 airmaster01 airflow[26451]: return dialect.connect(*cargs, > **cparams) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/sqlalchemy/engine/default.py", > line 393, in connect > May 21 09:51:58 airmaster01 airflow[26451]: return self.dbapi.connect(*cargs, > **cparams) > May 21 09:51:58 airmaster01 airflow[26451]: File > "/opt/airflow/production/lib/python3.4/site-packages/psycopg2/__init__.py", > line 130, in connect > May 21 09:51:58 airmaster01 airflow[26451]: conn = _connect(dsn, > connection_factory=connection_factory, **kwasync) > May 21 09:51:58 airmaster01 airflow[26451]: sqlalchemy.exc.OperationalError: > (psycopg2.OperationalError) could not translate host name > "airstorage01.example.com" to address: Name or service not known > May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:57 +0000] > [23794] [INFO] Worker exiting (pid: 23794) > May 21 09:51:58 airmaster01 airflow[26451]: > /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: > ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use > flask_cache instead. > May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), > ExtDeprecationWarning > May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] > [23561] [INFO] Worker exiting (pid: 23561) > May 21 09:51:58 airmaster01 airflow[26451]: > /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: > ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use > flask_cache instead. > May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), > ExtDeprecationWarning > May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] > [23759] [INFO] Worker exiting (pid: 23759) > May 21 09:51:58 airmaster01 airflow[26451]: > /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: > ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use > flask_cache instead. > May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), > ExtDeprecationWarning > May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] > [23620] [INFO] Worker exiting (pid: 23620) > May 21 09:51:58 airmaster01 airflow[26451]: > /opt/airflow/production/lib/python3.4/site-packages/flask/exthook.py:71: > ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use > flask_cache instead. > May 21 09:51:58 airmaster01 airflow[26451]: .format(x=modname), > ExtDeprecationWarning > May 21 09:51:58 airmaster01 airflow[26451]: [2017-05-21 09:51:58 +0000] > [23689] [INFO] Worker exiting (pid: 23689) > May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] > [26464] [INFO] Shutting down: Master > May 21 09:51:59 airmaster01 airflow[26451]: [2017-05-21 09:51:59 +0000] > [26464] [INFO] Reason: Worker failed to boot. > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)