cloventt commented on issue #23512:
URL: https://github.com/apache/airflow/issues/23512#issuecomment-1226669818
I see this same issue very frequently when the webserver starts up. Here are
logs of a crashed webserver pod:
```
[2022-08-25 01:30:38 +0000] [28] [INFO] Starting gunicorn 20.1.0
[2022-08-25 01:30:42,401] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
/home/airflow/.local/lib/python3.9/site-packages/azure/storage/common/_connection.py:82
SyntaxWarning: "is" with a literal. Did you mean "=="?
[2022-08-25 01:30:43 +0000] [28] [INFO] Listening at: http://0.0.0.0:8080
(28)
[2022-08-25 01:30:43 +0000] [28] [INFO] Using worker: sync
[2022-08-25 01:30:43 +0000] [46] [INFO] Booting worker with pid: 46
[2022-08-25 01:30:43 +0000] [47] [INFO] Booting worker with pid: 47
[2022-08-25 01:30:44 +0000] [48] [INFO] Booting worker with pid: 48
[2022-08-25 01:30:44 +0000] [49] [INFO] Booting worker with pid: 49
/home/airflow/.local/lib/python3.9/site-packages/airflow/www/app.py:111
DeprecationWarning: 'app.json_encoder' is deprecated and will be removed in
Flask 2.3. Customize 'app.json_provider_class' or 'app.json' instead.
/home/airflow/.local/lib/python3.9/site-packages/airflow/www/app.py:111
DeprecationWarning: 'app.json_encoder' is deprecated and will be removed in
Flask 2.3. Customize 'app.json_provider_class' or 'app.json' instead.
/home/airflow/.local/lib/python3.9/site-packages/airflow/www/app.py:111
DeprecationWarning: 'app.json_encoder' is deprecated and will be removed in
Flask 2.3. Customize 'app.json_provider_class' or 'app.json' instead.
/home/airflow/.local/lib/python3.9/site-packages/airflow/www/app.py:111
DeprecationWarning: 'app.json_encoder' is deprecated and will be removed in
Flask 2.3. Customize 'app.json_provider_class' or 'app.json' instead.
[2022-08-25 01:30:44,548] {manager.py:819} WARNING - No user yet created,
use flask fab command to do it.
[2022-08-25 01:30:44,596] {manager.py:819} WARNING - No user yet created,
use flask fab command to do it.
[2022-08-25 01:30:44,600] {manager.py:819} WARNING - No user yet created,
use flask fab command to do it.
[2022-08-25 01:30:44,611] {manager.py:819} WARNING - No user yet created,
use flask fab command to do it.
[2022-08-25 01:30:44,632] {manager.py:585} INFO - Removed Permission can
create on Users to role Admin
[2022-08-25 01:30:44,659] {manager.py:543} INFO - Removed Permission View:
can_create on Users
[2022-08-25 01:30:45,965] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:46,044] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:46,052] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:46,105] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:46,141] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:46,220] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:46,236] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:46,278] {providers_manager.py:215} INFO - Optional
provider feature disabled when importing
'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from
'apache-airflow-providers-google' package
[2022-08-25 01:30:47,298] {manager.py:508} INFO - Created Permission View:
can create on Users
[2022-08-25 01:30:47 +0000] [48] [ERROR] Exception in worker process
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1802, in _execute_context
self.dialect.do_execute(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 719, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: duplicate key value violates unique
constraint "ab_permission_view_role_permission_view_id_role_id_key"
DETAIL: Key (permission_view_id, role_id)=(27009, 1) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/gunicorn/arbiter.py", line
589, in spawn_worker
worker.init_process()
File
"/home/airflow/.local/lib/python3.9/site-packages/gunicorn/workers/base.py",
line 134, in init_process
self.load_wsgi()
File
"/home/airflow/.local/lib/python3.9/site-packages/gunicorn/workers/base.py",
line 146, in load_wsgi
self.wsgi = self.app.wsgi()
File
"/home/airflow/.local/lib/python3.9/site-packages/gunicorn/app/base.py", line
67, in wsgi
self.callable = self.load()
File
"/home/airflow/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py",
line 58, in load
return self.load_wsgiapp()
File
"/home/airflow/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py",
line 48, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/airflow/.local/lib/python3.9/site-packages/gunicorn/util.py",
line 412, in import_app
app = app(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/www/app.py", line
158, in cached_app
app = create_app(config=config, testing=testing)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/www/app.py", line
146, in create_app
sync_appbuilder_roles(flask_app)
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/www/app.py", line 68,
in sync_appbuilder_roles
flask_app.appbuilder.sm.sync_roles()
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/www/security.py",
line 596, in sync_roles
self.update_admin_permission()
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/www/security.py",
line 578, in update_admin_permission
self.get_session.commit()
File "<string>", line 2, in commit
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 1428, in commit
self._transaction.commit(_to_root=self.future)
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 829, in commit
self._prepare_impl()
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 808, in _prepare_impl
self.session.flush()
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3345, in flush
self._flush(objects)
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3485, in _flush
transaction.rollback(_capture_exception=True)
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",
line 70, in __exit__
compat.raise_(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 207, in raise_
raise exception
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py",
line 3445, in _flush
flush_context.execute()
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py",
line 456, in execute
rec.execute(self)
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py",
line 579, in execute
self.dependency_processor.process_saves(uow, states)
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/dependency.py",
line 1182, in process_saves
self._run_crud(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/dependency.py",
line 1245, in _run_crud
connection.execute(statement, secondary_insert)
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1289, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py",
line 325, in _execute_on_connection
return connection._execute_clauseelement(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1481, in _execute_clauseelement
ret = self._execute_context(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1845, in _execute_context
self._handle_dbapi_exception(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 2026, in _handle_dbapi_exception
util.raise_(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py",
line 207, in raise_
raise exception
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1802, in _execute_context
self.dialect.do_execute(
File
"/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py",
line 719, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate
key value violates unique constraint
"ab_permission_view_role_permission_view_id_role_id_key"
DETAIL: Key (permission_view_id, role_id)=(27009, 1) already exists.
[SQL: INSERT INTO ab_permission_view_role (id, permission_view_id, role_id)
VALUES (nextval('ab_permission_view_role_id_seq'), %(permission_view_id)s,
%(role_id)s) RETURNING ab_permission_view_role.id]
[parameters: {'permission_view_id': 27009, 'role_id': 1}]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
[2022-08-25 01:30:47 +0000] [48] [INFO] Worker exiting (pid: 48)
10.202.97.30 - - [25/Aug/2022:01:30:47 +0000] "GET /home HTTP/1.1" 200
341888 "-" "kube-probe/1.21+"
[2022-08-25 01:30:48 +0000] [46] [INFO] Worker exiting (pid: 46)
[2022-08-25 01:30:48 +0000] [47] [INFO] Worker exiting (pid: 47)
[2022-08-25 01:30:48 +0000] [49] [INFO] Worker exiting (pid: 49)
[2022-08-25 01:30:49 +0000] [28] [INFO] Shutting down: Master
[2022-08-25 01:30:49 +0000] [28] [INFO] Reason: Worker failed to boot.
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
[2022-08-25 01:30:50,071] {webserver_command.py:252} ERROR - [0 / 0] Some
workers seem to have died and gunicorn did not restart them as expected
[2022-08-25 01:31:00,078] {webserver_command.py:264} INFO - [0 / 0] Spawning
1 workers
[2022-08-25 01:33:00,177] {webserver_command.py:217} ERROR - No response
from gunicorn master within 120 seconds
[2022-08-25 01:33:00,177] {webserver_command.py:218} ERROR - Shutting down
webserver
```
I am running Airflow 2.3.4 on kubernetes. I have two webserver pods each
with 4 workers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
