santurini opened a new issue, #59279: URL: https://github.com/apache/airflow/issues/59279
### Apache Airflow version 3.1.3 ### If "Other Airflow 2/3 version" selected, which one? _No response_ ### What happened? I tried to upgrade my airflow helm chart from version 3.0.2 to version 3.1.3. I started with a fresh database and namespace, but when I try to access airflow at http://airflow.staging.mydomain.cloud I get redirected to http://airflow.staging.mydomain.cloud/auth/login/?next=http://airflow.staging.mydomain.cloud/ and it shows `Internal Server Error`. Inspecting the api-server logs I found this: ``` INFO: 10.42.5.9:48748 - "GET /ui/config HTTP/1.1" 401 Unauthorized INFO: 10.42.5.9:48754 - "GET /static/i18n/locales/en/common.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48748 - "GET /static/i18n/locales/en/dashboard.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48768 - "GET /static/i18n/locales/en/dags.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48786 - "GET /static/i18n/locales/en/assets.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48774 - "GET /static/i18n/locales/en/components.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48748 - "GET /static/i18n/locales/en/browse.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48754 - "GET /static/i18n/locales/en/admin.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48792 - "GET /static/i18n/locales/en/hitl.json HTTP/1.1" 304 Not Modified INFO: 10.42.5.9:48792 - "GET /api/v2/auth/login?next=http%3A%2F%2Fairflow.staging.translated.cloud%2F HTTP/1.1" 307 Temporary Redirect 2025-12-10T15:27:14.185556Z [error ] Exception in callback <function TaskGroup._spawn.<locals>.task_done at 0x74592c24fb00> handle: <Handle TaskGroup._spawn.<locals>.task_done> [asyncio] loc=runners.py:118 Traceback (most recent call last): File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 820, in task_done self._on_completed_fut.set_result(None) asyncio.exceptions.InvalidStateError: invalid state 2025-12-10T15:27:14.187211Z [error ] Exception in callback <function TaskGroup._spawn.<locals>.task_done at 0x74592c24eb60> handle: <Handle TaskGroup._spawn.<locals>.task_done> [asyncio] loc=runners.py:118 Traceback (most recent call last): File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 825, in task_done exc = _task.exception() ^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro await self.app(scope, receive_or_disconnect, send_no_error) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__ await self.middleware_stack(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app await route.handle(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 462, in handle await self.app(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1082, in __call__ await super().__call__(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__ await self.middleware_stack(scope, receive, send) File "/opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.11-manylinux2014/ddtrace/contrib/internal/asgi/middleware.py", line 456, in __call__ return await self.app(scope, wrapped_recv, wrapped_send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__ await self.middleware_stack(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app await route.handle(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/routing.py", line 462, in handle await self.app(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/wsgi.py", line 81, in __call__ await responder(receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/wsgi.py", line 101, in __call__ message = await receive() ^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 117, in receive_or_disconnect async with anyio.create_task_group() as task_group: File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 755, in __aexit__ await self._on_completed_fut asyncio.exceptions.CancelledError: Cancelled via cancel scope 74592c5d9cd0 by <Task pending name='starlette.middleware.base.BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.coro' coro=<BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.coro() running at /home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py:144> cb=[TaskGroup._spawn.<locals>.task_done() at /home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py:805]> INFO: 10.42.5.9:48792 - "GET /auth/login/?next=http://airflow.staging.translated.cloud/ HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application + Exception Group Traceback (most recent call last): | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_utils.py", line 79, in collapse_excgroups | yield | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 183, in __call__ | async with anyio.create_task_group() as task_group: | File "/home/airflow/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 783, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/airflow/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1082, in __call__ | await super().__call__(scope, receive, send) | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__ | await self.middleware_stack(scope, receive, send) | File "/opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.11-manylinux2014/ddtrace/contrib/internal/asgi/middleware.py", line 456, in __call__ | return await self.app(scope, wrapped_recv, wrapped_send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ | raise exc | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ | await self.app(scope, receive, _send) | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 29, in __call__ | await responder(scope, receive, send) | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 130, in __call__ | await super().__call__(scope, receive, send) | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 46, in __call__ | await self.app(scope, receive, self.send_with_compression) | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__ | with recv_stream, send_stream, collapse_excgroups(): | File "/usr/python/lib/python3.11/contextlib.py", line 158, in __exit__ | self.gen.throw(typ, value, traceback) | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_utils.py", line 85, in collapse_excgroups | raise exc | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__ | response = await self.dispatch_func(request, call_next) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/airflow/.local/lib/python3.11/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch | response = await call_next(request) | ^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 160, in call_next | raise RuntimeError("No response returned.") | RuntimeError: No response returned. +------------------------------------ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1082, in __call__ await super().__call__(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__ await self.middleware_stack(scope, receive, send) File "/opt/datadog/apm/library/python/ddtrace_pkgs/site-packages-ddtrace-py3.11-manylinux2014/ddtrace/contrib/internal/asgi/middleware.py", line 456, in __call__ return await self.app(scope, wrapped_recv, wrapped_send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 29, in __call__ await responder(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 130, in __call__ await super().__call__(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 46, in __call__ await self.app(scope, receive, self.send_with_compression) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__ with recv_stream, send_stream, collapse_excgroups(): File "/usr/python/lib/python3.11/contextlib.py", line 158, in __exit__ self.gen.throw(typ, value, traceback) File "/home/airflow/.local/lib/python3.11/site-packages/starlette/_utils.py", line 85, in collapse_excgroups raise exc File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__ response = await self.dispatch_func(request, call_next) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch response = await call_next(request) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/starlette/middleware/base.py", line 160, in call_next raise RuntimeError("No response returned.") RuntimeError: No response returned. ``` ### What you think should happen instead? I should be able to access the login page ### How to reproduce Custom docker image: ``` FROM apache/airflow:3.1.3-python3.11 USER root RUN apt-get update && \ apt-get install -y git g++ php php-cli php-common && \ apt-get clean && rm -rf /var/lib/apt/lists/* USER airflow ``` Helm values: ``` images: airflow: repository: ${IMAGE_NAME} tag: ${VERSION} pullPolicy: Always ingress: apiServer: enabled: true annotations: projectcontour.io/response-timeout: "120s" hosts: - name: "airflow.staging.mydomain.cloud" ingressClassName: "contour" env: - name: "AIRFLOW__API__AUTH_BACKEND" value: "airflow.api.auth.backend.basic_auth" - name: "AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION" value: "True" - name: "AIRFLOW__CORE__LOAD_EXAMPLES" value: "False" - name: AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER value: "True" - name: AIRFLOW__LOGGING__REMOTE_LOGGING value: "True" - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID value: "LoggingS3Connection" - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value: "s3://logs-archive-product-team/Airflow" - name: AIRFLOW__LOGGING__ENCRYPT_S3_LOGS value: "False" - name: AIRFLOW__CORE__ENABLE_XCOM_PICKLING value: "True" - name: AIRFLOW__SMTP__SMTP_HOST value: "smtp.gmail.com" - name: AIRFLOW__SMTP__SMTP_PORT value: "587" - name: AIRFLOW__SMTP__SMTP_SSL value: "False" - name: AIRFLOW__SMTP__SMTP_STARTTLS value: "True" - name: AIRFLOW__SMTP__SMTP_USER value: "[email protected]" - name: "CM_API_URL" value: "https://cm-api.mydomain.cloud/api"; secret: - envName: "GIT_TOKEN" secretName: "airflow-git-token" secretKey: "GIT_TOKEN" data: metadataSecretName: "airflow-pgbouncer-connection" apiSecretKey: ${API_SECRET_KEY} jwtSecret: ${JWT_SECRET} fernetKey: ${FERNET_KEY} webserverSecretKey: ${WEBSERVER_SECRET_KEY} workers: replicas: 1 registry: secretName: regcred postgresql: enabled: false pgbouncer: enabled: true configSecretName: airflow-pgbouncer-config metricsExporterSidecar: statsSecretName: airflow-pgbouncer-stats dags: gitSync: enabled: true repo: [email protected]:mydomain/airflow-dags.git branch: develop ref: develop depth: 1 subPath: "dags" sshKeySecret: airflow-ssh-secret ``` Deployment script: ``` export PGBOUNCER_INI_BASE64=$(envsubst < k8s/pgbouncer/pgbouncer.ini | base64 -w 0) export PGBOUNCER_USERS_BASE64=$(envsubst < k8s/pgbouncer/users.txt | base64 -w 0) envsubst <k8s/pgbouncer/airflow-pgbouncer-connection.yaml >airflow-pgbouncer-connection.yaml envsubst <k8s/pgbouncer/airflow-pgbouncer-stats.yaml >airflow-pgbouncer-stats.yaml envsubst <k8s/pgbouncer/airflow-pgbouncer-config.yaml >airflow-pgbouncer-config.yaml envsubst <k8s/airflow-aws-secrets.yaml >airflow-aws-secrets.yaml envsubst <k8s/airflow-connections-secrets.yaml >airflow-connections-secrets.yaml envsubst <k8s/airflow-git-secrets.yaml >airflow-git-secrets.yaml envsubst <k8s/airflow-git-token.yaml >airflow-git-token.yaml echo Installing secrets ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-aws-secrets.yaml ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-connections-secrets.yaml ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-git-secrets.yaml ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-git-token.yaml ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-pgbouncer-connection.yaml ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-pgbouncer-stats.yaml ssh -p 2222 docker-deploy@${SWARM_HOST} 'cat - | kubectl apply -f -' <airflow-pgbouncer-config.yaml echo Importing settings envsubst <k8s/${ENV}-airflow-values.yml >airflow-values.yml echo Airflow helm repo adding ssh -p 2222 docker-deploy@${SWARM_HOST} 'helm repo add apache-airflow https://airflow.apache.org' ssh -p 2222 docker-deploy@${SWARM_HOST} 'helm repo update' echo Airflow Helm installing ssh -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -p 2222 docker-deploy@${SWARM_HOST} "cat - | helm upgrade --install airflow apache-airflow/airflow --namespace airflow-$ENV -f -" <airflow-values.yml ``` ### Operating System Ubuntu 24.04.2 LTS ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details ``` images: airflow: repository: ${IMAGE_NAME} tag: ${VERSION} pullPolicy: Always ingress: apiServer: enabled: true annotations: projectcontour.io/response-timeout: "120s" hosts: - name: "airflow.staging.translated.cloud" ingressClassName: "contour" env: - name: "AIRFLOW__API__AUTH_BACKEND" value: "airflow.api.auth.backend.basic_auth" - name: "AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION" value: "True" - name: "AIRFLOW__CORE__LOAD_EXAMPLES" value: "False" - name: AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER value: "True" - name: AIRFLOW__LOGGING__REMOTE_LOGGING value: "True" - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID value: "LoggingS3Connection" - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value: "s3://logs-archive-product-team/Airflow" - name: AIRFLOW__LOGGING__ENCRYPT_S3_LOGS value: "False" - name: AIRFLOW__CORE__ENABLE_XCOM_PICKLING value: "True" - name: AIRFLOW__SMTP__SMTP_HOST value: "smtp.gmail.com" - name: AIRFLOW__SMTP__SMTP_PORT value: "587" - name: AIRFLOW__SMTP__SMTP_SSL value: "False" - name: AIRFLOW__SMTP__SMTP_STARTTLS value: "True" - name: AIRFLOW__SMTP__SMTP_USER value: "[email protected]" - name: "CM_API_URL" value: "https://cm-api.translated.cloud/api"; secret: - envName: "GIT_TOKEN" secretName: "airflow-git-token" secretKey: "GIT_TOKEN" data: metadataSecretName: "airflow-pgbouncer-connection" apiSecretKey: ${API_SECRET_KEY} jwtSecret: ${JWT_SECRET} fernetKey: ${FERNET_KEY} webserverSecretKey: ${WEBSERVER_SECRET_KEY} workers: replicas: 1 registry: secretName: regcred postgresql: enabled: false pgbouncer: enabled: true configSecretName: airflow-pgbouncer-config metricsExporterSidecar: statsSecretName: airflow-pgbouncer-stats dags: gitSync: enabled: true repo: [email protected]:translated/airflow-dags.git branch: develop ref: develop depth: 1 subPath: "dags" sshKeySecret: airflow-ssh-secret ``` ### Anything else? We were able to solve the issue by manually fixing the ingress definition who was pointing to the service `airflow-api-server:0`, by changing it to `airflow-api-server:8080` everything worked fine and the page stopped giving internal serror ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
