shivaam opened a new pull request, #64056: URL: https://github.com/apache/airflow/pull/64056
## Summary Fixes #56271 — KubernetesExecutor tasks fail with `UnknownExecutorException` in Airflow 3.1.0+. **Root cause:** `_validate_executor_fields()` in `DagBag.process_file()` runs during DAG parsing in ALL contexts, including inside KubernetesExecutor worker pods. Worker pods are configured with only `LocalExecutor` (via `pod_template_file`), so tasks specifying `executor='KubernetesExecutor'` fail validation even though they are valid. **Fix:** Add a `validate_executors` parameter to `DagBag` (default `True`) and set it to `False` in the task runner's `parse()` function. Worker pods don't need to validate executor assignments — that's the scheduler/dag-processor's responsibility. ## Changes | File | Change | |------|--------| | `airflow-core/src/airflow/dag_processing/dagbag.py` | Add `validate_executors` param to `DagBag.__init__`, skip `_validate_executor_fields()` when `False` | | `task-sdk/src/airflow/sdk/execution_time/task_runner.py` | Pass `validate_executors=False` to `BundleDagBag` in `parse()` | | `airflow-core/tests/unit/dag_processing/test_dagbag.py` | Add test verifying `validate_executors=False` skips validation | ## How it was before vs after | Context | Before (broken) | After (fixed) | |---------|-----------------|---------------| | Scheduler/dag-processor | Validates executors ✅ | Validates executors ✅ | | K8s worker pod | Validates → **FAILS** ❌ | Skips validation ✅ | | CLI (`airflow dags list`) | Validates ✅ | Validates ✅ | ## Test plan - [x] Unit test: `test_validate_executors_false_skips_validation` — DAG with unknown executor loads when `validate_executors=False`, fails when `True` - [ ] Integration: Deploy on K8s with multi-executor config, trigger task with `executor='KubernetesExecutor'` closes: #56271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
