GitHub user kardamk created a discussion: Airflow worker pod not able to fetch connections information during its execution
# Facing an issue while upgrading from Airflow 2.11.x to 3.1.8 hosted on OpenShift I keep getting the error, "httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'http://airflow-api-service:8080/execution/connections/aws_default"; when Airflow DAG is executed within a pod. I tested the connectivity from worker pod <-> api-server pod using curl and was able to connect successfully to the API server ruling out network connectivity issues. As suggested here https://github.com/apache/airflow/discussions/55380#discussioncomment-14781891, I already tried configuring the environment variable AIRFLOW__CORE__EXECUTION_API_SERVER_URL=http://airflow-api-service:8080/execution to no avail. Requesting for any inputs from the community to analyze and debug the issue further. ## Deployment Information Apache Airflow 3.1.8 deployed on OpenShift version 4.18.41 using official Airflow Helm chart version 1.19.0. Currently, migrating from 2.11.x to 3.1.8. Remote logging is enabled with an S3 compatible storage. Metadatabase is a Postgres crunchy cluster hosted on OpenShift within namespace. ## Apache Airflow version | 3.1.8 executor | KubernetesExecutor task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler sql_alchemy_conn | postgresql+psycopg2://xxxx dags_folder | /opt/airflow/dags/repo/dags plugins_folder | /opt/airflow/plugins base_log_folder | /opt/airflow/logs remote_base_log_folder | s3://xxx/airflow/yyyy/ ## System info OS | Linux architecture | x86_64 uname | uname_result(system='Linux', node='xxx, release='5.14.0-427.125.1.el9_4.x86_64', version='#1 SMP PREEMPT_DYNAMIC Sat May 9 17:31:37 EDT 2026', machine='x86_64') locale | ('C', 'UTF-8') python_version | 3.12.13 (main, Mar 11 2026, 20:29:10) [GCC 12.2.0] python_location | /usr/python/bin/python3.12 ## Tools info git | git version 2.39.5 ssh | OpenSSH_9.2p1 Debian-2+deb12u7, OpenSSL 3.0.18 30 Sep 2025 kubectl | NOT AVAILABLE gcloud | NOT AVAILABLE cloud_sql_proxy | NOT AVAILABLE mysql | mysql Ver 15.1 Distrib 10.11.16-MariaDB, for debian-linux-gnu (x86_64) using EditLine wrapper sqlite3 | 3.40.1 2022-12-28 14:03:47 df5c253c0b3dd24916e4ec7cf77d3db5294cc9fd45ae7b9c5e82ad8197f3alt1 psql | psql (PostgreSQL) 18.3 (Debian 18.3-1.pgdg12+1) ## Providers info apache-airflow-providers-amazon | 9.22.0 apache-airflow-providers-celery | 3.17.0 apache-airflow-providers-cncf-kubernetes | 10.8.2 apache-airflow-providers-common-compat | 1.14.0 apache-airflow-providers-common-io | 1.7.1 apache-airflow-providers-common-messaging | 2.0.2 apache-airflow-providers-common-sql | 1.30.4 apache-airflow-providers-docker | 4.5.2 apache-airflow-providers-elasticsearch | 6.5.0 apache-airflow-providers-fab | 3.4.0 apache-airflow-providers-ftp | 3.14.1 apache-airflow-providers-git | 0.2.4 apache-airflow-providers-google | 20.0.0 apache-airflow-providers-grpc | 3.9.2 apache-airflow-providers-hashicorp | 4.5.0 apache-airflow-providers-http | 6.0.0 apache-airflow-providers-microsoft-azure | 13.0.0 apache-airflow-providers-microsoft-mssql | 4.4.1 apache-airflow-providers-mysql | 6.5.0 apache-airflow-providers-odbc | 4.12.0 apache-airflow-providers-openlineage | 2.11.0 apache-airflow-providers-postgres | 6.6.0 apache-airflow-providers-redis | 4.4.2 apache-airflow-providers-samba | 4.12.2 apache-airflow-providers-sendgrid | 4.2.1 apache-airflow-providers-sftp | 5.7.0 apache-airflow-providers-slack | 9.7.0 apache-airflow-providers-smtp | 2.4.2 apache-airflow-providers-snowflake | 6.9.0 apache-airflow-providers-ssh | 4.3.1 apache-airflow-providers-standard | 1.12.0 ## Detailed error logs {"timestamp":"2026-07-10T19:31:45.305110Z","level":"info","event":"Connecting to server:","server":"http://airflow-api-service:8080/execution","logger":"__main__","filename":"execute_workload.py","lineno":64} {"timestamp":"2026-07-10T19:31:45.334611Z","level":"debug","event":"Connecting to execution API server","server":"http://airflow-api-service:8080/execution","logger":"supervisor","filename":"supervisor.py","lineno":1964} {"timestamp":"2026-07-10T19:31:45.338203Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.345337Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8165b20>","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.345626Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.345986Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.346139Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.346304Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.346415Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.348000Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.348756Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.348951Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:45.349252Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40} {"timestamp":"2026-07-10T19:31:46.350088Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.357830Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d81664b0>","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.358111Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.358405Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.358538Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.358684Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.358789Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.360397Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.360815Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.360991Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:46.361226Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40} {"timestamp":"2026-07-10T19:31:48.214846Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.222447Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8166ea0>","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.222715Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.223010Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.223155Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.223300Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.223642Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.225096Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.225478Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.225640Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:48.225821Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40} {"timestamp":"2026-07-10T19:31:51.414937Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.421926Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8166540>","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.422180Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.422440Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.422570Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.422712Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.422818Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.424488Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.424870Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.425034Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:51.425239Z","level":"warning","event":"Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it.","logger":"airflow.sdk.api.client","filename":"before.py","lineno":40} {"timestamp":"2026-07-10T19:31:55.267800Z","level":"debug","event":"connect_tcp.started host='internal-api-host' port=8080 local_address=None timeout=5.0 socket_options=None","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.277167Z","level":"debug","event":"connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7f11d8165af0>","logger":"httpcore.connection","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.277425Z","level":"debug","event":"send_request_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.277703Z","level":"debug","event":"send_request_headers.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.277836Z","level":"debug","event":"send_request_body.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.278030Z","level":"debug","event":"send_request_body.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.278178Z","level":"debug","event":"receive_response_headers.started request=<Request [b'GET']>","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.280298Z","level":"debug","event":"receive_response_headers.complete return_value=(b'HTTP/1.0', 503, b'Service Unavailable', [(b'Connection', b'close')])","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.280656Z","level":"debug","event":"response_closed.started","logger":"httpcore.http11","filename":"_trace.py","lineno":47} {"timestamp":"2026-07-10T19:31:55.280816Z","level":"debug","event":"response_closed.complete","logger":"httpcore.http11","filename":"_trace.py","lineno":47} Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 125, in <module> main() File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 121, in main execute_workload(workload) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py", line 66, in execute_workload supervise( File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1972, in supervise logger, log_file_descriptor = _configure_logging(log_path, client) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1887, in _configure_logging with _remote_logging_conn(client): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 899, in _remote_logging_conn conn = _fetch_remote_logging_conn(conn_id, client) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 863, in _fetch_remote_logging_conn conn = client.connections.get(conn_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 361, in get resp = self.client.get(f"connections/{conn_id}") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get return self.request( ^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f return copy(f, *args, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter result = action(retry_state) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 413, in exc_check raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 184, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__ result = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request return super().request(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send response = self._send_handling_auth( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth response = self._send_handling_redirects( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects raise exc File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects hook(response) File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 186, in raise_on_4xx_5xx_with_note return get_json_error(response) or response.raise_for_status() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_models.py", line 829, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'http://airflow-api-service:8080/execution/connections/aws_default' GitHub link: https://github.com/apache/airflow/discussions/69733 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
