amoghrajesh commented on code in PR #57601:
URL: https://github.com/apache/airflow/pull/57601#discussion_r2480101859
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
Review Comment:
There's no webserver in Airflow 3!
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
Review Comment:
Not a valid command anymore:
```python
root@4ea09d3b5d7a:/opt/airflow# airflow tasks test afexception
afexception_task --help
root@4ea09d3b5d7a:/opt/airflow# airflow tasks test afexception
afexception_task --help
/opt/airflow/airflow-core/src/airflow/dag_processing/dagbag.py:40
DeprecationWarning: airflow.exceptions.AirflowDagCycleException is deprecated.
Use airflow.sdk.exceptions.AirflowDagCycleException instead.
Usage: airflow tasks test [-h] [-B BUNDLE_NAME] [--env-vars ENV_VARS]
[--map-index MAP_INDEX] [-m]
[-t TASK_PARAMS] [-v]
dag_id task_id [logical_date_or_run_id]
Test a task instance. This will run a task without checking for dependencies
or recording its state in the database
Positional Arguments:
dag_id The id of the dag
task_id The id of the task
logical_date_or_run_id
The logical date of the DAG or run_id of the DAGRun
(optional)
```
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
+- Monitor system resources during task execution
+- Check connection pool settings and limits
+- Review retry configuration and implement exponential backoff
+
+Rule 3: Quit Thinking and Look
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Examine actual logs and data rather than making assumptions.
+
+**Essential Airflow Log Locations:**
+
+- Task logs: ``$AIRFLOW_HOME/logs/dag_id/task_id/execution_date/``
+- Scheduler logs: ``$AIRFLOW_HOME/logs/scheduler/``
+- Webserver logs: Check your webserver configuration
+- Worker logs: ``airflow celery worker`` output
Review Comment:
What does this provide?
```python
root@4ea09d3b5d7a:/opt/airflow# airflow celery worker
/opt/airflow/airflow-core/src/airflow/dag_processing/dagbag.py:40
DeprecationWarning: airflow.exceptions.AirflowDagCycleException is deprecated.
Use airflow.sdk.exceptions.AirflowDagCycleException instead.
2025-10-31T03:58:33.109627Z [info ] starting stale bundle cleanup
process [airflow.providers.celery.cli.celery_command] loc=celery_command.py:141
2025-10-31T03:58:33.113733Z [info ] Starting log server on
http://[::]:8793 [airflow.utils.serve_logs.core] loc=core.py:5
```
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
+- Monitor system resources during task execution
+- Check connection pool settings and limits
+- Review retry configuration and implement exponential backoff
+
+Rule 3: Quit Thinking and Look
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Examine actual logs and data rather than making assumptions.
+
+**Essential Airflow Log Locations:**
+
+- Task logs: ``$AIRFLOW_HOME/logs/dag_id/task_id/execution_date/``
Review Comment:
This is wrong too, format is:
```
logs/dag_id=abcd/run_id=manual__2025-07-29T08:44:37.608526+00:0/task_id=say_hello
```
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
+- Monitor system resources during task execution
+- Check connection pool settings and limits
+- Review retry configuration and implement exponential backoff
+
+Rule 3: Quit Thinking and Look
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Examine actual logs and data rather than making assumptions.
+
+**Essential Airflow Log Locations:**
+
+- Task logs: ``$AIRFLOW_HOME/logs/dag_id/task_id/execution_date/``
+- Scheduler logs: ``$AIRFLOW_HOME/logs/scheduler/``
+- Webserver logs: Check your webserver configuration
+- Worker logs: ``airflow celery worker`` output
+
+**Example: "Task Not Found" Error**
+
+*Symptoms:*
+- Error: "Task 'task_id' not found in DAG 'dag_id'"
+- Task exists in DAG file
+- Other tasks in same DAG work fine
+
+*Diagnosis:*
+Instead of assuming the task definition is correct, examine the actual parsed
DAG.
+
+*Suggestions:*
+- Check DAG parsing: ``airflow dags show dag_id``
+- Verify task_id spelling and case sensitivity
+- Look for conditional task creation logic
+- Check if task is dynamically generated and conditions are met
+
+Rule 4: Divide and Conquer
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Isolate problems by testing components separately.
+
+**Airflow Isolation Strategies:**
+
+- Test individual tasks: ``airflow tasks test``
+- Test connections: ``airflow connections test``
+- Test DAG parsing: ``airflow dags list-import-errors``
+- Test database connectivity: ``airflow db check``
+
+**Example: DAG Import Failures**
+
+*Symptoms:*
+- DAGs not appearing in UI
+- Import errors in scheduler logs
+- Some DAGs work, others don't
+
+*Diagnosis:*
+Isolate the problematic DAG from working ones to identify the specific issue.
+
+*Suggestions:*
+- Test DAG parsing individually: ``python /path/to/dag.py``
+- Check import errors: ``airflow dags list-import-errors``
+- Move problematic DAG to separate directory for testing
+- Verify all imports and dependencies are available
+
+Rule 5: Change One Thing at a Time
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Make incremental changes to avoid introducing new problems.
+
+**Airflow Change Management:**
+
+- Modify one configuration parameter at a time
+- Test single task changes before applying to entire DAG
+- Update one dependency version at a time
+- Change one environment variable per test cycle
+
+**Example: Performance Optimization**
+
+*Symptoms:*
+- Slow DAG execution
+- Tasks taking longer than expected
+- Resource utilization issues
+
+*Diagnosis:*
+Multiple performance factors could be involved.
+
+*Suggestions:*
+- Change one setting at a time: parallelism, pool slots, or worker count
+- Test each change with consistent workload
+- Monitor metrics after each modification
+- Document performance impact of each change
+
+Rule 6: Keep an Audit Trail
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Document your debugging process and changes made.
+
+**Airflow Debugging Documentation:**
+
+- Log configuration changes with timestamps
+- Record error messages and their contexts
+- Document successful and failed solutions
+- Track performance metrics before/after changes
+
+**Example: Connection Configuration Issues**
+
+*Symptoms:*
+- Tasks fail with authentication errors
+- Connection tests pass in UI
+- Intermittent connection failures
+
+*Diagnosis:*
+Connection configuration might have subtle issues not apparent in simple tests.
+
+*Suggestions:*
+- Document exact connection parameters tested
+- Record which authentication methods were tried
+- Log environment variables and their values
+- Keep track of successful connection configurations
+
+Rule 7: Check the Plug
+^^^^^^^^^^^^^^^^^^^^^^
+
+Verify basic assumptions and simple causes first.
+
+**Airflow "Plug" Checks:**
+
+- Is Airflow running? ``airflow version``
+- Are services accessible? ``airflow db check``
+- Are DAGs in the correct directory? ``echo $AIRFLOW_HOME/dags``
+- Are permissions correct? ``ls -la $AIRFLOW_HOME/dags``
+
+**Example: DAGs Not Loading**
+
+*Symptoms:*
+- No DAGs visible in UI
+- Scheduler appears to be running
+- No obvious error messages
+
+*Diagnosis:*
+Before investigating complex issues, check basic requirements.
+
+*Suggestions:*
+- Verify DAG directory path: ``airflow config get-value core dags_folder``
+- Check file permissions and ownership
+- Confirm DAG files have ``.py`` extension
+- Test with a simple example DAG
+
+Rule 8: Get a Fresh View
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Seek external perspectives when stuck.
+
+**Airflow Community Resources:**
+
+- `Apache Airflow Slack <https://s.apache.org/airflow-slack>`_
+- `GitHub Issues <https://github.com/apache/airflow/issues>`_
+- `Stack Overflow <https://stackoverflow.com/questions/tagged/airflow>`_
+- `Airflow Documentation <https://airflow.apache.org/docs/>`_
+
+**Example: Complex DAG Dependencies**
+
+*Symptoms:*
+- DAG runs but tasks execute in wrong order
+- Dependencies seem correct in code
+- Graph view shows unexpected relationships
+
+*Diagnosis:*
+Complex dependency logic might have subtle issues not obvious to the original
author.
+
+*Suggestions:*
+- Ask colleague to review DAG structure
+- Post dependency graph on community forums
+- Compare with similar working DAGs
+- Use ``airflow tasks list dag_id --tree`` to visualize dependencies
Review Comment:
This command does not exist!
```
airflow command error: unrecognized arguments: --tree, see help above.
```
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
+- Monitor system resources during task execution
+- Check connection pool settings and limits
+- Review retry configuration and implement exponential backoff
+
+Rule 3: Quit Thinking and Look
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Examine actual logs and data rather than making assumptions.
+
+**Essential Airflow Log Locations:**
+
+- Task logs: ``$AIRFLOW_HOME/logs/dag_id/task_id/execution_date/``
+- Scheduler logs: ``$AIRFLOW_HOME/logs/scheduler/``
+- Webserver logs: Check your webserver configuration
Review Comment:
No webserver exists.
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
+- Monitor system resources during task execution
+- Check connection pool settings and limits
+- Review retry configuration and implement exponential backoff
+
+Rule 3: Quit Thinking and Look
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Examine actual logs and data rather than making assumptions.
+
+**Essential Airflow Log Locations:**
+
+- Task logs: ``$AIRFLOW_HOME/logs/dag_id/task_id/execution_date/``
+- Scheduler logs: ``$AIRFLOW_HOME/logs/scheduler/``
+- Webserver logs: Check your webserver configuration
+- Worker logs: ``airflow celery worker`` output
+
+**Example: "Task Not Found" Error**
+
+*Symptoms:*
+- Error: "Task 'task_id' not found in DAG 'dag_id'"
+- Task exists in DAG file
+- Other tasks in same DAG work fine
+
+*Diagnosis:*
+Instead of assuming the task definition is correct, examine the actual parsed
DAG.
+
+*Suggestions:*
+- Check DAG parsing: ``airflow dags show dag_id``
+- Verify task_id spelling and case sensitivity
+- Look for conditional task creation logic
+- Check if task is dynamically generated and conditions are met
+
+Rule 4: Divide and Conquer
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Isolate problems by testing components separately.
+
+**Airflow Isolation Strategies:**
+
+- Test individual tasks: ``airflow tasks test``
+- Test connections: ``airflow connections test``
+- Test DAG parsing: ``airflow dags list-import-errors``
+- Test database connectivity: ``airflow db check``
+
+**Example: DAG Import Failures**
+
+*Symptoms:*
+- DAGs not appearing in UI
+- Import errors in scheduler logs
Review Comment:
We have a dag processor that parses dags independently in Airflow 3, why
would you see it in here?
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
+- Monitor system resources during task execution
+- Check connection pool settings and limits
+- Review retry configuration and implement exponential backoff
+
+Rule 3: Quit Thinking and Look
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Examine actual logs and data rather than making assumptions.
+
+**Essential Airflow Log Locations:**
+
+- Task logs: ``$AIRFLOW_HOME/logs/dag_id/task_id/execution_date/``
+- Scheduler logs: ``$AIRFLOW_HOME/logs/scheduler/``
+- Webserver logs: Check your webserver configuration
+- Worker logs: ``airflow celery worker`` output
+
+**Example: "Task Not Found" Error**
+
+*Symptoms:*
+- Error: "Task 'task_id' not found in DAG 'dag_id'"
+- Task exists in DAG file
+- Other tasks in same DAG work fine
+
+*Diagnosis:*
+Instead of assuming the task definition is correct, examine the actual parsed
DAG.
+
+*Suggestions:*
+- Check DAG parsing: ``airflow dags show dag_id``
+- Verify task_id spelling and case sensitivity
+- Look for conditional task creation logic
+- Check if task is dynamically generated and conditions are met
+
+Rule 4: Divide and Conquer
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Isolate problems by testing components separately.
+
+**Airflow Isolation Strategies:**
+
+- Test individual tasks: ``airflow tasks test``
+- Test connections: ``airflow connections test``
+- Test DAG parsing: ``airflow dags list-import-errors``
+- Test database connectivity: ``airflow db check``
+
+**Example: DAG Import Failures**
+
+*Symptoms:*
+- DAGs not appearing in UI
+- Import errors in scheduler logs
+- Some DAGs work, others don't
+
+*Diagnosis:*
+Isolate the problematic DAG from working ones to identify the specific issue.
+
+*Suggestions:*
+- Test DAG parsing individually: ``python /path/to/dag.py``
Review Comment:
In which component? Have you tested this?
##########
airflow-core/docs/troubleshooting.rst:
##########
@@ -17,52 +17,454 @@
.. _troubleshooting:
-Troubleshooting
-===============
+How to Debug Your Airflow Deployment
+====================================
-Obscure task failures
+This guide provides a systematic approach to debugging Airflow deployments
based on proven debugging principles. It's designed to help you think like an
Airflow debugger and develop effective troubleshooting strategies for your
specific deployment.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+What You Should Expect from This Guide
+--------------------------------------
+
+**What this guide provides:**
+
+- A systematic debugging methodology for Airflow deployments
+- Real-world examples with symptoms, diagnosis, and suggestions
+- Guidelines for building your own debugging practices
+- Links to relevant documentation and community resources
+
+**What this guide does NOT provide:**
+
+- Comprehensive solutions for every possible Airflow issue
+- End-to-end recipes for all deployment scenarios
+- Guaranteed fixes for complex infrastructure problems
+
+**Your responsibility:**
+
+- Adapt these guidelines to your specific deployment environment
+- Build deployment-specific debugging procedures
+- Maintain logs and monitoring appropriate for your setup
+- Understand your infrastructure and dependencies
+
+The 9 Rules of Airflow Debugging
+--------------------------------
+
+Based on Agans' debugging principles, here's how to approach Airflow issues
systematically:
+
+Rule 1: Understand the System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before debugging, understand your Airflow architecture and components.
+
+**Key Airflow Components to Understand:**
+
+- **Scheduler**: Orchestrates DAG execution and task scheduling
+- **Executor**: Manages task execution (LocalExecutor, CeleryExecutor,
KubernetesExecutor)
+- **Webserver**: Provides the UI and API
+- **Workers**: Execute tasks (in distributed setups)
+- **Database**: Stores metadata and state
+- **Message Broker**: Queues tasks (Redis/RabbitMQ for Celery)
+
+**Example: Task Stuck in Queued State**
+
+*Symptoms:*
+- Tasks remain in "queued" state indefinitely
+- No error messages in task logs
+- DAG appears to run normally
+
+*Diagnosis:*
+Understanding the system helps identify that queued tasks indicate an executor
issue - tasks are scheduled but not picked up for execution.
+
+*Suggestions:*
+- Check executor configuration and worker availability
+- Verify message broker connectivity (for CeleryExecutor)
+- Review resource limits and worker capacity
+- Check ``airflow celery worker`` logs for distributed setups
+
+Rule 2: Make It Fail
+^^^^^^^^^^^^^^^^^^^^
+
+Reproduce issues consistently to understand their patterns.
+
+**Airflow-Specific Reproduction Strategies:**
+
+- Test with minimal DAGs to isolate issues
+- Use ``airflow tasks test`` for individual task debugging
+- Trigger DAG runs manually to control timing
+- Modify task configurations incrementally
+
+**Example: Intermittent Task Failures**
+
+*Symptoms:*
+- Tasks fail randomly with connection timeouts
+- Same task succeeds on retry
+- No clear pattern in failure timing
+
+*Diagnosis:*
+Intermittent failures often indicate resource contention, network issues, or
race conditions.
+
+*Suggestions:*
+- Run the task multiple times: ``airflow tasks test dag_id task_id
execution_date``
+- Monitor system resources during task execution
+- Check connection pool settings and limits
+- Review retry configuration and implement exponential backoff
+
+Rule 3: Quit Thinking and Look
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Examine actual logs and data rather than making assumptions.
+
+**Essential Airflow Log Locations:**
+
+- Task logs: ``$AIRFLOW_HOME/logs/dag_id/task_id/execution_date/``
+- Scheduler logs: ``$AIRFLOW_HOME/logs/scheduler/``
+- Webserver logs: Check your webserver configuration
+- Worker logs: ``airflow celery worker`` output
+
+**Example: "Task Not Found" Error**
+
+*Symptoms:*
+- Error: "Task 'task_id' not found in DAG 'dag_id'"
+- Task exists in DAG file
+- Other tasks in same DAG work fine
+
+*Diagnosis:*
+Instead of assuming the task definition is correct, examine the actual parsed
DAG.
+
+*Suggestions:*
+- Check DAG parsing: ``airflow dags show dag_id``
+- Verify task_id spelling and case sensitivity
+- Look for conditional task creation logic
+- Check if task is dynamically generated and conditions are met
+
+Rule 4: Divide and Conquer
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Isolate problems by testing components separately.
+
+**Airflow Isolation Strategies:**
+
+- Test individual tasks: ``airflow tasks test``
+- Test connections: ``airflow connections test``
+- Test DAG parsing: ``airflow dags list-import-errors``
+- Test database connectivity: ``airflow db check``
+
+**Example: DAG Import Failures**
+
+*Symptoms:*
+- DAGs not appearing in UI
+- Import errors in scheduler logs
+- Some DAGs work, others don't
+
+*Diagnosis:*
+Isolate the problematic DAG from working ones to identify the specific issue.
+
+*Suggestions:*
+- Test DAG parsing individually: ``python /path/to/dag.py``
+- Check import errors: ``airflow dags list-import-errors``
+- Move problematic DAG to separate directory for testing
+- Verify all imports and dependencies are available
+
+Rule 5: Change One Thing at a Time
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Make incremental changes to avoid introducing new problems.
+
+**Airflow Change Management:**
+
+- Modify one configuration parameter at a time
+- Test single task changes before applying to entire DAG
+- Update one dependency version at a time
+- Change one environment variable per test cycle
+
+**Example: Performance Optimization**
+
+*Symptoms:*
+- Slow DAG execution
+- Tasks taking longer than expected
+- Resource utilization issues
+
+*Diagnosis:*
+Multiple performance factors could be involved.
+
+*Suggestions:*
+- Change one setting at a time: parallelism, pool slots, or worker count
+- Test each change with consistent workload
+- Monitor metrics after each modification
+- Document performance impact of each change
+
+Rule 6: Keep an Audit Trail
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Document your debugging process and changes made.
+
+**Airflow Debugging Documentation:**
+
+- Log configuration changes with timestamps
+- Record error messages and their contexts
+- Document successful and failed solutions
+- Track performance metrics before/after changes
+
+**Example: Connection Configuration Issues**
+
+*Symptoms:*
+- Tasks fail with authentication errors
+- Connection tests pass in UI
+- Intermittent connection failures
+
+*Diagnosis:*
+Connection configuration might have subtle issues not apparent in simple tests.
+
+*Suggestions:*
+- Document exact connection parameters tested
+- Record which authentication methods were tried
+- Log environment variables and their values
+- Keep track of successful connection configurations
+
+Rule 7: Check the Plug
+^^^^^^^^^^^^^^^^^^^^^^
+
+Verify basic assumptions and simple causes first.
+
+**Airflow "Plug" Checks:**
+
+- Is Airflow running? ``airflow version``
+- Are services accessible? ``airflow db check``
+- Are DAGs in the correct directory? ``echo $AIRFLOW_HOME/dags``
+- Are permissions correct? ``ls -la $AIRFLOW_HOME/dags``
Review Comment:
We tell to do certain things here, but whats the expectation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
