GitHub user Ewald-Hoffranzen created a discussion: [Airflow 3.1.7] Task
execution fails with "Invalid auth token" using LocalExecutor and Task SDK
### Apache Airflow version
3.1.7
### If "Other Airflow 3 version" selected, which one?
_No response_
### What happened?
### Apache Airflow version
3.1.7
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Task execution consistently fails with `Invalid auth token: Signature
verification failed` despite Scheduler and API-Server having identical JWT
secrets.
### What you think should happen instead?
Tasks should execute successfully with valid JWT authentication between
Scheduler and API-Server.
### How to reproduce
**Environment:**
- Airflow: 3.1.7 (Docker image: apache/airflow:3.1.7-python3.12)
- Executor: LocalExecutor
- Database: PostgreSQL 15
- Deployment: Docker Compose
**Configuration:**
```yaml
services:
airflow-scheduler:
environment:
AIRFLOW__CORE__INTERNAL_API_URL: "http://airflow-apiserver:8080";
AIRFLOW__CORE__EXECUTION_API_SERVER_URL:
"http://airflow-apiserver:8080/execution/";
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
AIRFLOW__CORE__EXECUTOR: "LocalExecutor"
airflow-apiserver:
environment:
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
```
**Steps:**
1. Deploy Airflow 3.1.7 with above configuration
2. Verify both services have identical JWT secrets: `airflow config get-value
secrets jwt_secret`
3. Trigger any DAG task
4. Observe immediate failure
### Operating System
Ubuntu 22.04 (Docker)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
Docker Compose with separate containers for:
- airflow-apiserver
- airflow-scheduler
- airflow-dag-processor
- airflow-triggerer
- PostgreSQL (external, version 15)
### Anything else?
**Error Logs:**
Scheduler error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 966, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature
verification failed
Correlation-id=019c6702-76ee-7344-b98b-b3890d1e64df
```
API-Server error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py",
line 328, in avalidated_claims
self._verify_signature(signing_input, header, signature, key, algorithms)
jwt.exceptions.InvalidSignatureError: Signature verification failed
```
**Observations:**
1. Both services confirmed to have identical JWT secrets (86 bytes,
SHA512-compliant)
2. Problem persists after deleting jwt_secret from database and complete
service recreation
3. `InsecureKeyLengthWarning` appears intermittently, suggesting worker
subprocess uses cached/different secret
4. Setting `AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth` is
**ignored** by Task SDK
5. Worker subprocess in LocalExecutor creates tokens that API-Server cannot
validate
**Suspected Root Cause:**
Worker subprocess inherits or caches a different JWT secret than API-Server
uses for validation, possibly due to:
- Forked process not inheriting updated environment variables
- Task SDK hardcoded to use JWT regardless of auth_backends configuration
- Race condition in secret loading during worker subprocess creation
**Impact:**
Critical - Airflow 3.1.7 with LocalExecutor cannot execute any tasks.
**Workarounds Attempted:**
- ✗ Recreating all containers
- ✗ Deleting jwt_secret from database
- ✗ Setting Basic Auth (ignored by Task SDK)
- ✗ Ensuring identical secrets across all services
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
### What you think should happen instead?
_No response_
### How to reproduce
### Apache Airflow version
3.1.7
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Task execution consistently fails with `Invalid auth token: Signature
verification failed` despite Scheduler and API-Server having identical JWT
secrets.
### What you think should happen instead?
Tasks should execute successfully with valid JWT authentication between
Scheduler and API-Server.
### How to reproduce
**Environment:**
- Airflow: 3.1.7 (Docker image: apache/airflow:3.1.7-python3.12)
- Executor: LocalExecutor
- Database: PostgreSQL 15
- Deployment: Docker Compose
**Configuration:**
```yaml
services:
airflow-scheduler:
environment:
AIRFLOW__CORE__INTERNAL_API_URL: "http://airflow-apiserver:8080";
AIRFLOW__CORE__EXECUTION_API_SERVER_URL:
"http://airflow-apiserver:8080/execution/";
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
AIRFLOW__CORE__EXECUTOR: "LocalExecutor"
airflow-apiserver:
environment:
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
```
**Steps:**
1. Deploy Airflow 3.1.7 with above configuration
2. Verify both services have identical JWT secrets: `airflow config get-value
secrets jwt_secret`
3. Trigger any DAG task
4. Observe immediate failure
### Operating System
Ubuntu 22.04 (Docker)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
Docker Compose with separate containers for:
- airflow-apiserver
- airflow-scheduler
- airflow-dag-processor
- airflow-triggerer
- PostgreSQL (external, version 15)
### Anything else?
**Error Logs:**
Scheduler error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 966, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature
verification failed
Correlation-id=019c6702-76ee-7344-b98b-b3890d1e64df
```
API-Server error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py",
line 328, in avalidated_claims
self._verify_signature(signing_input, header, signature, key, algorithms)
jwt.exceptions.InvalidSignatureError: Signature verification failed
```
**Observations:**
1. Both services confirmed to have identical JWT secrets (86 bytes,
SHA512-compliant)
2. Problem persists after deleting jwt_secret from database and complete
service recreation
3. `InsecureKeyLengthWarning` appears intermittently, suggesting worker
subprocess uses cached/different secret
4. Setting `AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth` is
**ignored** by Task SDK
5. Worker subprocess in LocalExecutor creates tokens that API-Server cannot
validate
**Suspected Root Cause:**
Worker subprocess inherits or caches a different JWT secret than API-Server
uses for validation, possibly due to:
- Forked process not inheriting updated environment variables
- Task SDK hardcoded to use JWT regardless of auth_backends configuration
- Race condition in secret loading during worker subprocess creation
**Impact:**
Critical - Airflow 3.1.7 with LocalExecutor cannot execute any tasks.
**Workarounds Attempted:**
- ✗ Recreating all containers
- ✗ Deleting jwt_secret from database
- ✗ Setting Basic Auth (ignored by Task SDK)
- ✗ Ensuring identical secrets across all services
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
### Operating System
ubuntu-20.04.3-live-server-amd64
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/62057
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
