This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 55d648ec094 use of debugpy to run airflow components via breeze
(#51763)
55d648ec094 is described below
commit 55d648ec094fc6f22a28f34bef0656a062705494
Author: Kalyan R <[email protected]>
AuthorDate: Thu Jul 24 17:05:57 2025 +0530
use of debugpy to run airflow components via breeze (#51763)
* debug support in breeze
* fix celery env var
* Add debug options for Airflow components and update related constants
* Add debug options for Airflow components and update related constants
* refactor
* update documentation and add setup_vscode
* Remove section on overriding default debug ports in VSCode setup
documentation
* fix base-ports
* fix base-ports
* update docs
* add webserver details in docs
* update docs
* update docs
* update docs
---
contributing-docs/06_development_environments.rst | 3 +
.../20_debugging_airflow_components.rst | 162 +++++++++++++++++++++
contributing-docs/README.rst | 3 +
dev/breeze/doc/03_developer_tasks.rst | 6 +
dev/breeze/doc/images/output_start-airflow.svg | 40 ++++-
dev/breeze/doc/images/output_start-airflow.txt | 2 +-
.../src/airflow_breeze/commands/common_options.py | 19 +++
.../airflow_breeze/commands/developer_commands.py | 8 +
.../commands/developer_commands_config.py | 4 +
dev/breeze/src/airflow_breeze/global_constants.py | 7 +
.../src/airflow_breeze/params/shell_params.py | 42 ++++++
scripts/ci/docker-compose/base-ports.yml | 7 +
scripts/in_container/bin/run_tmux | 48 +++++-
setup_vscode.py | 129 ++++++++++++++++
14 files changed, 468 insertions(+), 12 deletions(-)
diff --git a/contributing-docs/06_development_environments.rst
b/contributing-docs/06_development_environments.rst
index 285f87d99eb..5ce138a3f1c 100644
--- a/contributing-docs/06_development_environments.rst
+++ b/contributing-docs/06_development_environments.rst
@@ -158,3 +158,6 @@ Typically, you are recommended to use multiple of these
environments depending o
If you want to learn more details about setting up your local virtualenv,
follow to the
`Local virtualenv <07_local_virtualenv.rst>`__ document.
+
+For detailed information about debugging Airflow components using Breeze, see
the
+`Debugging Airflow Components <20_debugging_airflow_components.rst>`__ guide.
diff --git a/contributing-docs/20_debugging_airflow_components.rst
b/contributing-docs/20_debugging_airflow_components.rst
new file mode 100644
index 00000000000..c6502f787fb
--- /dev/null
+++ b/contributing-docs/20_debugging_airflow_components.rst
@@ -0,0 +1,162 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Debugging Airflow Components with Breeze
+========================================
+
+Breeze provides debugging support for Airflow components using the ``--debug``
and ``--debugger`` flags
+in the ``breeze start-airflow`` command.
+
+Starting Airflow with Debug Support
+----------------------------------
+
+To start Airflow with debugging enabled, use the ``--debug`` flag to specify
which components you want to debug:
+
+.. code-block:: bash
+
+ # Debug the scheduler
+ breeze start-airflow --debug scheduler
+
+ # Debug multiple components
+ breeze start-airflow --debug scheduler --debug triggerer
+
+ # Debug all components
+ breeze start-airflow --debug scheduler --debug triggerer --debug
api-server --debug dag-processor
+
+ # Debug with CeleryExecutor
+ breeze start-airflow -b postgres -P 17 --executor CeleryExecutor --debug
scheduler --debug dag-processor --debug api-server --debug triggerer --debug
celery-worker
+
+ # Debug Webserver for Airflow 2.x
+ breeze start-airflow --debug webserver
+
+Available Components for Debugging
+----------------------------------
+
+* **scheduler** - The Airflow scheduler that monitors DAGs and triggers task
instances
+* **triggerer** - The triggerer service that handles deferred tasks and
triggers
+* **api-server** - The Airflow REST API server
+* **dag-processor** - The DAG processor service (when using standalone DAG
processor)
+* **edge-worker** - The edge worker service (when using EdgeExecutor)
+* **celery-worker** - Celery worker processes (when using CeleryExecutor)
+
+Debugger Options
+----------------
+
+Breeze supports two debugger options:
+
+* **debugpy** (default)
+* **pydevd-pycharm**
+
+.. code-block:: bash
+
+ # Use debugpy (default)
+ breeze start-airflow --debug scheduler --debugger debugpy
+
+ # Use PyCharm debugger
+ breeze start-airflow --debug scheduler --debugger pydevd-pycharm
+
+Setting up VSCode for Remote Debugging
+--------------------------------------
+
+1. **Install Required Extensions**
+
+ Install the following VSCode extensions:
+ * Python (ms-python.python)
+ * Python Debugger (ms-python.debugpy)
+
+2. **Create Launch Configuration**
+
+ Create or update your ``.vscode/launch.json`` file. The easiest way is to
run the setup script:
+
+ .. code-block:: bash
+
+ python setup_vscode.py
+
+ This will create debug configurations for all Airflow components. Here's an
example configuration for the scheduler:
+
+ .. code-block:: json
+
+ {
+ "name": "Debug Airflow Scheduler",
+ "type": "debugpy",
+ "request": "attach",
+ "justMyCode": false,
+ "connect": {
+ "host": "localhost",
+ "port": 50231
+ },
+ "pathMappings": [
+ {
+ "localRoot": "${workspaceFolder}",
+ "remoteRoot": "/opt/airflow"
+ }
+ ]
+ }
+
+3. **Port Mapping**
+
+ Each component uses a different debug port. These ports are automatically
assigned by Breeze
+ when you start Airflow with debugging enabled:
+
+ * **Scheduler**: 50231
+ * **DAG Processor**: 50232
+ * **Triggerer**: 50233
+ * **API Server**: 50234
+ * **Celery Worker**: 50235
+ * **Edge Worker**: 50236
+ * **Web Server**: 50237
+
+ These ports are exposed from the Breeze container to your host machine,
allowing your IDE
+ to connect to the debugger running inside the container.
+
+
+Debugging Workflow
+------------------
+
+1. **Start Airflow with Debug Support**
+
+ .. code-block:: bash
+
+ breeze start-airflow --debug scheduler --debugger debugpy
+
+2. **Set Breakpoints**
+
+ In VSCode, set breakpoints in your Airflow code by clicking in the gutter
next to line numbers.
+
+3. **Attach Debugger**
+
+ - Open the Debug panel in VSCode (Ctrl+Shift+D / Cmd+Shift+D)
+ - Select the appropriate debug configuration (e.g., "Debug Airflow
Scheduler")
+ - Click the green play button or press F5
+
+4. **Trigger Debugging**
+
+ Perform an action that will trigger the code path with your breakpoint:
+
+ - For scheduler: Trigger a DAG or wait for scheduled execution
+ - For API server: Make an API call
+ - For triggerer: Create a deferred task
+ - For DAG processor: Parse a DAG file
+
+5. **Debug Session**
+
+ Once the breakpoint is hit:
+
+ - Inspect variables in the Variables panel
+ - Use the Debug Console to evaluate expressions
+ - Step through code using F10 (step over), F11 (step into), F12 (step out)
+ - Continue execution with F5
diff --git a/contributing-docs/README.rst b/contributing-docs/README.rst
index ebe192b0e05..c58a3089b42 100644
--- a/contributing-docs/README.rst
+++ b/contributing-docs/README.rst
@@ -121,3 +121,6 @@ You can also dive deeper into more specific areas that are
important for contrib
* `Execution API versioning <19_execution_api_versioning.rst>`__ describes how
to
version the Task Execution API and how to add new versions of the API.
+
+* `Debugging Airflow Components <20_debugging_airflow_components.rst>`__
describes how to debug
+ Airflow components using Breeze with debugpy and VSCode integration.
diff --git a/dev/breeze/doc/03_developer_tasks.rst
b/dev/breeze/doc/03_developer_tasks.rst
index d269e98a0c8..fd8530684b4 100644
--- a/dev/breeze/doc/03_developer_tasks.rst
+++ b/dev/breeze/doc/03_developer_tasks.rst
@@ -193,6 +193,12 @@ your local sources to the ``/opt/airflow`` location of the
sources within the co
:align: center
:alt: Source code mapping
+.. note::
+
+ For comprehensive debugging documentation using the new ``--debug`` and
``--debugger`` flags
+ with VSCode and debugpy, see the `Debugging Airflow Components
<../../contributing-docs/20_debugging_airflow_components.rst>`__
+ guide.
+
Building the documentation
--------------------------
diff --git a/dev/breeze/doc/images/output_start-airflow.svg
b/dev/breeze/doc/images/output_start-airflow.svg
index 2d59895ce72..bf172e819a5 100644
--- a/dev/breeze/doc/images/output_start-airflow.svg
+++ b/dev/breeze/doc/images/output_start-airflow.svg
@@ -1,4 +1,4 @@
-<svg class="rich-terminal" viewBox="0 0 1482 3514.7999999999997"
xmlns="http://www.w3.org/2000/svg";>
+<svg class="rich-terminal" viewBox="0 0 1482 3661.2"
xmlns="http://www.w3.org/2000/svg";>
<!-- Generated with Rich https://www.textualize.io -->
<style>
@@ -43,7 +43,7 @@
<defs>
<clipPath id="breeze-start-airflow-clip-terminal">
- <rect x="0" y="0" width="1463.0" height="3463.7999999999997" />
+ <rect x="0" y="0" width="1463.0" height="3610.2" />
</clipPath>
<clipPath id="breeze-start-airflow-line-0">
<rect x="0" y="1.5" width="1464" height="24.65"/>
@@ -468,9 +468,27 @@
<clipPath id="breeze-start-airflow-line-140">
<rect x="0" y="3417.5" width="1464" height="24.65"/>
</clipPath>
+<clipPath id="breeze-start-airflow-line-141">
+ <rect x="0" y="3441.9" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-start-airflow-line-142">
+ <rect x="0" y="3466.3" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-start-airflow-line-143">
+ <rect x="0" y="3490.7" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-start-airflow-line-144">
+ <rect x="0" y="3515.1" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-start-airflow-line-145">
+ <rect x="0" y="3539.5" width="1464" height="24.65"/>
+ </clipPath>
+<clipPath id="breeze-start-airflow-line-146">
+ <rect x="0" y="3563.9" width="1464" height="24.65"/>
+ </clipPath>
</defs>
- <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="3512.8" rx="8"/><text
class="breeze-start-airflow-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: start-airflow</text>
+ <rect fill="#292929" stroke="rgba(255,255,255,0.35)" stroke-width="1"
x="1" y="1" width="1480" height="3659.2" rx="8"/><text
class="breeze-start-airflow-title" fill="#c5c8c6" text-anchor="middle" x="740"
y="27">Command: start-airflow</text>
<g transform="translate(26,22)">
<circle cx="0" cy="0" r="7" fill="#ff5f57"/>
<circle cx="22" cy="0" r="7" fill="#febc2e"/>
@@ -616,12 +634,18 @@
</text><text class="breeze-start-airflow-r5" x="0" y="3265.2"
textLength="24.4"
clip-path="url(#breeze-start-airflow-line-133)">╭─</text><text
class="breeze-start-airflow-r5" x="24.4" y="3265.2" textLength="183"
clip-path="url(#breeze-start-airflow-line-133)"> Other options </text><text
class="breeze-start-airflow-r5" x="207.4" y="3265.2" textLength="1232.2"
clip-path="url(#breeze-start-airflow-line-133)">────────────────────────────────────────────────────────────────────
[...]
</text><text class="breeze-start-airflow-r5" x="0" y="3289.6"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-134)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3289.6" textLength="256.2"
clip-path="url(#breeze-start-airflow-line-134)">--forward-credentials</text><text
class="breeze-start-airflow-r6" x="305" y="3289.6" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-134)">-f</text><text
class="breeze-start-airflow-r1" x="353.8" y="3289.6" textLength=" [...]
</text><text class="breeze-start-airflow-r5" x="0" y="3314" textLength="1464"
clip-path="url(#breeze-start-airflow-line-135)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-start-airflow-r1" x="1464" y="3314" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-135)">
-</text><text class="breeze-start-airflow-r5" x="0" y="3338.4"
textLength="24.4"
clip-path="url(#breeze-start-airflow-line-136)">╭─</text><text
class="breeze-start-airflow-r5" x="24.4" y="3338.4" textLength="195.2"
clip-path="url(#breeze-start-airflow-line-136)"> Common options </text><text
class="breeze-start-airflow-r5" x="219.6" y="3338.4" textLength="1220"
clip-path="url(#breeze-start-airflow-line-136)">───────────────────────────────────────────────────────────────────
[...]
-</text><text class="breeze-start-airflow-r5" x="0" y="3362.8"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-137)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3362.8" textLength="97.6"
clip-path="url(#breeze-start-airflow-line-137)">--answer</text><text
class="breeze-start-airflow-r6" x="158.6" y="3362.8" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-137)">-a</text><text
class="breeze-start-airflow-r1" x="207.4" y="3362.8" textLength="317.2" clip-
[...]
-</text><text class="breeze-start-airflow-r5" x="0" y="3387.2"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-138)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3387.2" textLength="109.8"
clip-path="url(#breeze-start-airflow-line-138)">--dry-run</text><text
class="breeze-start-airflow-r6" x="158.6" y="3387.2" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-138)">-D</text><text
class="breeze-start-airflow-r1" x="207.4" y="3387.2" textLength="719.8" cli
[...]
-</text><text class="breeze-start-airflow-r5" x="0" y="3411.6"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-139)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3411.6" textLength="109.8"
clip-path="url(#breeze-start-airflow-line-139)">--verbose</text><text
class="breeze-start-airflow-r6" x="158.6" y="3411.6" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-139)">-v</text><text
class="breeze-start-airflow-r1" x="207.4" y="3411.6" textLength="585.6" cli
[...]
-</text><text class="breeze-start-airflow-r5" x="0" y="3436" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-140)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3436" textLength="73.2"
clip-path="url(#breeze-start-airflow-line-140)">--help</text><text
class="breeze-start-airflow-r6" x="158.6" y="3436" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-140)">-h</text><text
class="breeze-start-airflow-r1" x="207.4" y="3436" textLength="329.4"
clip-path="url( [...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3338.4"
textLength="24.4"
clip-path="url(#breeze-start-airflow-line-136)">╭─</text><text
class="breeze-start-airflow-r5" x="24.4" y="3338.4" textLength="231.8"
clip-path="url(#breeze-start-airflow-line-136)"> Debugging options </text><text
class="breeze-start-airflow-r5" x="256.2" y="3338.4" textLength="1183.4"
clip-path="url(#breeze-start-airflow-line-136)">──────────────────────────────────────────────────────────────
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3362.8"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-137)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3362.8" textLength="85.4"
clip-path="url(#breeze-start-airflow-line-137)">--debug</text><text
class="breeze-start-airflow-r1" x="195.2" y="3362.8" textLength="1244.4"
clip-path="url(#breeze-start-airflow-line-137)">Enable debugging for specific Airflow components. Can be
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3387.2"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-138)">│</text><text
class="breeze-start-airflow-r1" x="195.2" y="3387.2" textLength="1244.4"
clip-path="url(#breeze-start-airflow-line-138)">api-server, dag-processor, edge-worker, celery-worker.                          
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3411.6"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-139)">│</text><text
class="breeze-start-airflow-r7" x="195.2" y="3411.6" textLength="1244.4"
clip-path="url(#breeze-start-airflow-line-139)">(scheduler | triggerer | api-server | dag-processor | edge-worker | celery-worker)                &
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3436" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-140)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3436" textLength="122"
clip-path="url(#breeze-start-airflow-line-140)">--debugger</text><text
class="breeze-start-airflow-r1" x="195.2" y="3436" textLength="597.8"
clip-path="url(#breeze-start-airflow-line-140)">Debugger to use for debugging Airflow components.</text><text
clas [...]
</text><text class="breeze-start-airflow-r5" x="0" y="3460.4"
textLength="1464"
clip-path="url(#breeze-start-airflow-line-141)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-start-airflow-r1" x="1464" y="3460.4" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-141)">
+</text><text class="breeze-start-airflow-r5" x="0" y="3484.8"
textLength="24.4"
clip-path="url(#breeze-start-airflow-line-142)">╭─</text><text
class="breeze-start-airflow-r5" x="24.4" y="3484.8" textLength="195.2"
clip-path="url(#breeze-start-airflow-line-142)"> Common options </text><text
class="breeze-start-airflow-r5" x="219.6" y="3484.8" textLength="1220"
clip-path="url(#breeze-start-airflow-line-142)">───────────────────────────────────────────────────────────────────
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3509.2"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-143)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3509.2" textLength="97.6"
clip-path="url(#breeze-start-airflow-line-143)">--answer</text><text
class="breeze-start-airflow-r6" x="158.6" y="3509.2" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-143)">-a</text><text
class="breeze-start-airflow-r1" x="207.4" y="3509.2" textLength="317.2" clip-
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3533.6"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-144)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3533.6" textLength="109.8"
clip-path="url(#breeze-start-airflow-line-144)">--dry-run</text><text
class="breeze-start-airflow-r6" x="158.6" y="3533.6" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-144)">-D</text><text
class="breeze-start-airflow-r1" x="207.4" y="3533.6" textLength="719.8" cli
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3558" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-145)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3558" textLength="109.8"
clip-path="url(#breeze-start-airflow-line-145)">--verbose</text><text
class="breeze-start-airflow-r6" x="158.6" y="3558" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-145)">-v</text><text
class="breeze-start-airflow-r1" x="207.4" y="3558" textLength="585.6"
clip-path=" [...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3582.4"
textLength="12.2" clip-path="url(#breeze-start-airflow-line-146)">│</text><text
class="breeze-start-airflow-r4" x="24.4" y="3582.4" textLength="73.2"
clip-path="url(#breeze-start-airflow-line-146)">--help</text><text
class="breeze-start-airflow-r6" x="158.6" y="3582.4" textLength="24.4"
clip-path="url(#breeze-start-airflow-line-146)">-h</text><text
class="breeze-start-airflow-r1" x="207.4" y="3582.4" textLength="329.4" clip-pa
[...]
+</text><text class="breeze-start-airflow-r5" x="0" y="3606.8"
textLength="1464"
clip-path="url(#breeze-start-airflow-line-147)">╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</text><text
class="breeze-start-airflow-r1" x="1464" y="3606.8" textLength="12.2"
clip-path="url(#breeze-start-airflow-line-147)">
</text>
</g>
</g>
diff --git a/dev/breeze/doc/images/output_start-airflow.txt
b/dev/breeze/doc/images/output_start-airflow.txt
index ad332f31642..063b275e3ba 100644
--- a/dev/breeze/doc/images/output_start-airflow.txt
+++ b/dev/breeze/doc/images/output_start-airflow.txt
@@ -1 +1 @@
-6a6bd606f4cbcec80a6335b1c16fcddc
+95c9bb3df25c755c805283de1424b9bb
diff --git a/dev/breeze/src/airflow_breeze/commands/common_options.py
b/dev/breeze/src/airflow_breeze/commands/common_options.py
index 15a5cbbce2e..e323801320a 100644
--- a/dev/breeze/src/airflow_breeze/commands/common_options.py
+++ b/dev/breeze/src/airflow_breeze/commands/common_options.py
@@ -477,6 +477,25 @@ option_install_airflow_with_constraints_default_true =
click.option(
envvar="INSTALL_AIRFLOW_WITH_CONSTRAINTS",
help="Install airflow in a separate step, with constraints determined from
package or airflow version.",
)
+option_debug_components = click.option(
+ "--debug",
+ "debug_components",
+ help="Enable debugging for specific Airflow components. Can be one or more
of: "
+ "scheduler, triggerer, api-server, dag-processor, edge-worker,
celery-worker.",
+ type=BetterChoice(
+ ["scheduler", "triggerer", "api-server", "dag-processor",
"edge-worker", "celery-worker"]
+ ),
+ multiple=True,
+ envvar="DEBUG_COMPONENTS",
+)
+option_debugger = click.option(
+ "--debugger",
+ help="Debugger to use for debugging Airflow components.",
+ type=BetterChoice(["debugpy", "pydevd-pycharm"]),
+ default="debugpy",
+ show_default=True,
+ envvar="DEBUGGER",
+)
def _is_number_greater_than_expected(value: str) -> bool:
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index b3b2b96af71..3b59bded514 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -41,6 +41,8 @@ from airflow_breeze.commands.common_options import (
option_builder,
option_clean_airflow_installation,
option_db_reset,
+ option_debug_components,
+ option_debugger,
option_docker_host,
option_downgrade_pendulum,
option_downgrade_sqlalchemy,
@@ -504,6 +506,8 @@ option_auth_manager_start_airflow = click.option(
@option_celery_broker
@option_celery_flower
@option_db_reset
+@option_debug_components
+@option_debugger
@option_docker_host
@option_dry_run
@option_executor_start_airflow
@@ -548,6 +552,8 @@ def start_airflow(
celery_flower: bool,
clean_airflow_installation: bool,
db_reset: bool,
+ debug_components: tuple[str, ...],
+ debugger: str,
dev_mode: bool,
docker_host: str | None,
executor: str | None,
@@ -617,6 +623,8 @@ def start_airflow(
celery_broker=celery_broker,
celery_flower=celery_flower,
clean_airflow_installation=clean_airflow_installation,
+ debug_components=debug_components,
+ debugger=debugger,
db_reset=db_reset,
dev_mode=dev_mode,
docker_host=docker_host,
diff --git
a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
index dfb01ff83ed..b4332f7c1a6 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands_config.py
@@ -293,6 +293,10 @@ DEVELOPER_PARAMETERS: dict[str, list[dict[str, str |
list[str]]]] = {
"--forward-credentials",
],
},
+ {
+ "name": "Debugging options",
+ "options": ["--debug", "--debugger"],
+ },
],
"breeze exec": [
{"name": "Drops in the interactive shell of active airflow container"},
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py
b/dev/breeze/src/airflow_breeze/global_constants.py
index c640cf34210..25e13a26c39 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -381,6 +381,13 @@ REDIS_HOST_PORT = "26379"
SSH_PORT = "12322"
VITE_DEV_PORT = "5173"
WEB_HOST_PORT = "28080"
+BREEZE_DEBUG_SCHEDULER_PORT = "50231"
+BREEZE_DEBUG_DAG_PROCESSOR_PORT = "50232"
+BREEZE_DEBUG_TRIGGERER_PORT = "50233"
+BREEZE_DEBUG_APISERVER_PORT = "50234"
+BREEZE_DEBUG_CELERY_WORKER_PORT = "50235"
+BREEZE_DEBUG_EDGE_PORT = "50236"
+BREEZE_DEBUG_WEBSERVER_PORT = "50237"
CELERY_BROKER_URLS_MAP = {"rabbitmq": "amqp://guest:guest@rabbitmq:5672",
"redis": "redis://redis:6379/0"}
SQLITE_URL = "sqlite:////root/airflow/sqlite/airflow.db"
diff --git a/dev/breeze/src/airflow_breeze/params/shell_params.py
b/dev/breeze/src/airflow_breeze/params/shell_params.py
index b0fee42cea5..f6bb0bfe174 100644
--- a/dev/breeze/src/airflow_breeze/params/shell_params.py
+++ b/dev/breeze/src/airflow_breeze/params/shell_params.py
@@ -37,6 +37,13 @@ from airflow_breeze.global_constants import (
ALLOWED_POSTGRES_VERSIONS,
ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
+ BREEZE_DEBUG_APISERVER_PORT,
+ BREEZE_DEBUG_CELERY_WORKER_PORT,
+ BREEZE_DEBUG_DAG_PROCESSOR_PORT,
+ BREEZE_DEBUG_EDGE_PORT,
+ BREEZE_DEBUG_SCHEDULER_PORT,
+ BREEZE_DEBUG_TRIGGERER_PORT,
+ BREEZE_DEBUG_WEBSERVER_PORT,
CELERY_BROKER_URLS_MAP,
CELERY_EXECUTOR,
DEFAULT_CELERY_BROKER,
@@ -149,6 +156,8 @@ class ShellParams:
celery_flower: bool = False
clean_airflow_installation: bool = False
collect_only: bool = False
+ debug_components: tuple[str, ...] = ()
+ debugger: str = "debugpy"
db_reset: bool = False
default_constraints_branch: str = DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH
dev_mode: bool = False
@@ -654,12 +663,45 @@ class ShellParams:
_set_var(_env, "WEB_HOST_PORT", None, WEB_HOST_PORT)
_set_var(_env, "_AIRFLOW_RUN_DB_TESTS_ONLY", self.run_db_tests_only)
_set_var(_env, "_AIRFLOW_SKIP_DB_TESTS", self.skip_db_tests)
+
+ self._set_debug_variables(_env)
self._generate_env_for_docker_compose_file_if_needed(_env)
_target_env: dict[str, str] = os.environ.copy()
_target_env.update(_env)
return _target_env
+ def _set_debug_variables(self, env) -> None:
+ """Set debug environment variables based on selected debug
components."""
+ if self.debugger == "pydevd-pycharm":
+ print("Pycharm-pydevd debugger is under development and not yet
supported in Breeze.")
+ return
+ if not self.debug_components:
+ return
+
+ _set_var(env, "BREEZE_DEBUG_SCHEDULER_PORT", None,
BREEZE_DEBUG_SCHEDULER_PORT)
+ _set_var(env, "BREEZE_DEBUG_DAG_PROCESSOR_PORT", None,
BREEZE_DEBUG_DAG_PROCESSOR_PORT)
+ _set_var(env, "BREEZE_DEBUG_TRIGGERER_PORT", None,
BREEZE_DEBUG_TRIGGERER_PORT)
+ _set_var(env, "BREEZE_DEBUG_APISERVER_PORT", None,
BREEZE_DEBUG_APISERVER_PORT)
+ _set_var(env, "BREEZE_DEBUG_CELERY_WORKER_PORT", None,
BREEZE_DEBUG_CELERY_WORKER_PORT)
+ _set_var(env, "BREEZE_DEBUG_EDGE_PORT", None, BREEZE_DEBUG_EDGE_PORT)
+ _set_var(env, "BREEZE_DEBUG_WEBSERVER_PORT", None,
BREEZE_DEBUG_WEBSERVER_PORT)
+
+ _set_var(env, "BREEZE_DEBUGGER", None, self.debugger)
+
+ component_mappings = {
+ "scheduler": "BREEZE_DEBUG_SCHEDULER",
+ "triggerer": "BREEZE_DEBUG_TRIGGERER",
+ "api-server": "BREEZE_DEBUG_APISERVER",
+ "dag-processor": "BREEZE_DEBUG_DAG_PROCESSOR",
+ "edge-worker": "BREEZE_DEBUG_EDGE",
+ "celery-worker": "BREEZE_DEBUG_CELERY_WORKER",
+ }
+
+ for component in self.debug_components:
+ env_var = component_mappings[component]
+ _set_var(env, env_var, None, "true")
+
@staticmethod
def _generate_env_for_docker_compose_file_if_needed(env: dict[str, str]):
"""
diff --git a/scripts/ci/docker-compose/base-ports.yml
b/scripts/ci/docker-compose/base-ports.yml
index d05771106ea..1e711563896 100644
--- a/scripts/ci/docker-compose/base-ports.yml
+++ b/scripts/ci/docker-compose/base-ports.yml
@@ -21,3 +21,10 @@ services:
- "${SSH_PORT}:22"
- "${WEB_HOST_PORT}:8080"
- "${FLOWER_HOST_PORT}:5555"
+ - "${BREEZE_DEBUG_SCHEDULER_PORT}:50231"
+ - "${BREEZE_DEBUG_DAG_PROCESSOR_PORT}:50232"
+ - "${BREEZE_DEBUG_TRIGGERER_PORT}:50233"
+ - "${BREEZE_DEBUG_APISERVER_PORT}:50234"
+ - "${BREEZE_DEBUG_CELERY_WORKER_PORT}:50235"
+ - "${BREEZE_DEBUG_EDGE_PORT}:50236"
+ - "${BREEZE_DEBUG_WEBSERVER_PORT}:50237"
diff --git a/scripts/in_container/bin/run_tmux
b/scripts/in_container/bin/run_tmux
index 431fe3faefc..60367737344 100755
--- a/scripts/in_container/bin/run_tmux
+++ b/scripts/in_container/bin/run_tmux
@@ -24,7 +24,7 @@ if [[ ${BACKEND} != "sqlite" ]]; then
export AIRFLOW__CORE__EXECUTOR=${AIRFLOW__CORE__EXECUTOR:-LocalExecutor}
fi
-#this is because I run docker in WSL - Hi Nadella!
+
export TMUX_TMPDIR=~/.tmux/tmp
if [ -e ~/.tmux/tmp ]; then
rm -rf ~/.tmux/tmp
@@ -57,22 +57,35 @@ tmux send-keys -t 'Main' 'bash' C-m 'clear' C-m
tmux split-window -v
tmux select-pane -t 1
+if [[ ${BREEZE_DEBUG_SCHEDULER} == "true" ]]; then
+ tmux set-option -p @airflow_component "Scheduler(Debug Mode)"
+ tmux send-keys "debugpy --listen 0.0.0.0:${BREEZE_DEBUG_SCHEDULER_PORT}
--wait-for-client -m airflow scheduler" C-m
+else
tmux set-option -p @airflow_component Scheduler
-tmux send-keys 'airflow scheduler' C-m
+ tmux send-keys "airflow scheduler" C-m
+fi
if [[ ! ${USE_AIRFLOW_VERSION=} =~ ^2\..* ]]; then
tmux split-window -h
tmux select-pane -t 2
+ if [[ ${BREEZE_DEBUG_APISERVER} == "true" ]]; then
+ tmux set-option -p @airflow_component "API Server(Debug Mode)"
+ tmux send-keys "debugpy --listen
0.0.0.0:${BREEZE_DEBUG_APISERVER_PORT} --wait-for-client -m airflow api-server
-d" C-m
+ else
tmux set-option -p @airflow_component "API Server"
-
if [[ ${DEV_MODE=} == "true" ]]; then
tmux send-keys 'airflow api-server -d' C-m
else
tmux send-keys 'airflow api-server' C-m
fi
+ fi
else
tmux split-window -h
tmux select-pane -t 3
+ if [[ ${BREEZE_DEBUG_WEBSERVER} == "true" ]]; then
+ tmux set-option -p @airflow_component "Webserver(Debug Mode)"
+ tmux send-keys "debugpy --listen
0.0.0.0:${BREEZE_DEBUG_WEBSERVER_PORT} --wait-for-client -m airflow webserver"
C-m
+ else
tmux set-option -p @airflow_component Webserver
if [[ ${DEV_MODE=} == "true" ]]; then
tmux send-keys 'airflow webserver -d' C-m
@@ -80,27 +93,49 @@ else
tmux send-keys 'airflow webserver' C-m
fi
fi
+fi
tmux select-pane -t 0
tmux split-window -h
+if [[ ${BREEZE_DEBUG_TRIGGERER} == "true" ]]; then
+ tmux set-option -p @airflow_component "Triggerer(Debug Mode)"
+ tmux send-keys "debugpy --listen 0.0.0.0:${BREEZE_DEBUG_TRIGGERER_PORT}
--wait-for-client -m airflow triggerer" C-m
+else
tmux set-option -p @airflow_component Triggerer
tmux send-keys 'airflow triggerer' C-m
+fi
if [[ ${INTEGRATION_CELERY} == "true" ]]; then
tmux select-pane -t 0
tmux split-window -h
+ if [[ ${BREEZE_DEBUG_CELERY_WORKER} == "true" ]]; then
+ tmux set-option -p @airflow_component "Celery Worker(Debug Mode)"
+ tmux send-keys "debugpy --listen
0.0.0.0:${BREEZE_DEBUG_CELERY_WORKER_PORT} --wait-for-client -m airflow celery
worker" C-m
+ else
tmux set-option -p @airflow_component "Celery Worker"
tmux send-keys 'airflow celery worker' C-m
fi
+fi
+
if [[ ${INTEGRATION_CELERY} == "true" && ${CELERY_FLOWER} == "true" ]]; then
tmux select-pane -t 3
tmux split-window -h
+ if [[ ${BREEZE_DEBUG_FLOWER} == "true" ]]; then
+ tmux set-option -p @airflow_component "Flower(Debug Mode)"
+ tmux send-keys "debugpy --listen 0.0.0.0:${BREEZE_DEBUG_FLOWER_PORT}
--wait-for-client -m airflow celery flower" C-m
+ else
tmux set-option -p @airflow_component Flower
tmux send-keys 'airflow celery flower' C-m
fi
+fi
+
if [[ ${AIRFLOW__CORE__EXECUTOR} ==
"airflow.providers.edge3.executors.edge_executor.EdgeExecutor" ]]; then
tmux select-pane -t 0
tmux split-window -h
+ if [[ ${BREEZE_DEBUG_EDGE} == "true" ]]; then
+ tmux set-option -p @airflow_component "Edge Worker(Debug Mode)"
+ tmux send-keys "debugpy --listen 0.0.0.0:${BREEZE_DEBUG_EDGE_PORT}
--wait-for-client -m airflow edge worker --edge-hostname breeze --queues
default" C-m
+ else
tmux set-option -p @airflow_component "Edge Worker"
# Ensure we are not leaking any DB connection information to Edge Worker
process
@@ -116,12 +151,19 @@ if [[ ${AIRFLOW__CORE__EXECUTOR} ==
"airflow.providers.edge3.executors.edge_exec
# Start Edge Worker and make a "breeze" hostname, let it pick only default
queue
tmux send-keys 'airflow edge worker --edge-hostname breeze --queues
default' C-m
fi
+fi
+
if [[ ${STANDALONE_DAG_PROCESSOR} == "true" ]]; then
tmux select-pane -t 3
tmux split-window -h
+ if [[ ${BREEZE_DEBUG_DAG_PROCESSOR} == "true" ]]; then
+ tmux set-option -p @airflow_component "DAG Processor(Debug Mode)"
+ tmux send-keys "debugpy --listen
0.0.0.0:${BREEZE_DEBUG_DAG_PROCESSOR_PORT} --wait-for-client -m airflow
dag-processor" C-m
+ else
tmux set-option -p @airflow_component "DAG Processor"
tmux send-keys 'airflow dag-processor' C-m
fi
+fi
# Attach Session, on the Main window
tmux select-pane -t 0
diff --git a/setup_vscode.py b/setup_vscode.py
new file mode 100755
index 00000000000..56b53f6fd93
--- /dev/null
+++ b/setup_vscode.py
@@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# ]
+# ///
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from rich import print
+from rich.prompt import Confirm
+
+# Debug port mappings for Airflow components
+DEBUG_PORTS = {
+ "scheduler": 50231,
+ "dag-processor": 50232,
+ "triggerer": 50233,
+ "api-server": 50234,
+ "celery-worker": 50235,
+ "edge-worker": 50236,
+}
+
+# Component descriptions for better naming
+COMPONENT_NAMES = {
+ "scheduler": "Scheduler",
+ "dag-processor": "DAG Processor",
+ "triggerer": "Triggerer",
+ "api-server": "API Server",
+ "celery-worker": "Celery Worker",
+ "edge-worker": "Edge Worker",
+}
+
+ROOT_AIRFLOW_FOLDER_PATH = Path(__file__).parent
+VSCODE_FOLDER_PATH = ROOT_AIRFLOW_FOLDER_PATH / ".vscode"
+LAUNCH_JSON_FILE = VSCODE_FOLDER_PATH / "launch.json"
+
+
+def create_debug_configuration(component: str, port: int) -> dict:
+ """Create a debug configuration for a specific Airflow component."""
+ return {
+ "name": f"Debug Airflow {COMPONENT_NAMES[component]}",
+ "type": "debugpy",
+ "request": "attach",
+ "justMyCode": False,
+ "connect": {"host": "localhost", "port": port},
+ "pathMappings": [{"localRoot": "${workspaceFolder}", "remoteRoot":
"/opt/airflow"}],
+ }
+
+
+def create_launch_json_content() -> dict:
+ """Create the complete launch.json content with all debug
configurations."""
+ configurations = []
+
+ for component, port in DEBUG_PORTS.items():
+ config = create_debug_configuration(component, port)
+ configurations.append(config)
+
+ return {"version": "0.2.0", "configurations": configurations}
+
+
+def setup_vscode():
+ """Set up VSCode debug configurations for Airflow components."""
+ print("[green]Creating[/] VSCode debug configurations for Airflow
components...")
+
+ # Create configurations for each component
+ for component, port in DEBUG_PORTS.items():
+ print(f"[green]Adding[/] debug configuration:
[blue]{COMPONENT_NAMES[component]}[/] (port {port})")
+
+ # Create the launch.json content
+ launch_json_content = create_launch_json_content()
+
+ # Ensure .vscode directory exists
+ VSCODE_FOLDER_PATH.mkdir(exist_ok=True)
+
+ # Write the launch.json file
+ with open(LAUNCH_JSON_FILE, "w") as f:
+ json.dump(launch_json_content, f, indent=4)
+
+ print(f"\n[green]Successfully created[/] {LAUNCH_JSON_FILE}")
+
+
+def main():
+ print("\n[yellow]VSCode Airflow Debug Configuration Setup[/]\n")
+ print("This script will create VSCode debug configurations for Airflow
components:\n")
+
+ for component, port in DEBUG_PORTS.items():
+ print(f"* {COMPONENT_NAMES[component]}: port {port}")
+
+ print(f"\nConfiguration will be written to: {LAUNCH_JSON_FILE}")
+
+ if LAUNCH_JSON_FILE.exists():
+ print(f"\n[yellow]Warning:[/] {LAUNCH_JSON_FILE} already exists!")
+ should_overwrite = Confirm.ask("Overwrite the existing file?")
+ if not should_overwrite:
+ print("[yellow]Skipped[/] - No changes made")
+ return
+ else:
+ should_continue = Confirm.ask("Create the debug configurations?")
+ if not should_continue:
+ print("[yellow]Skipped[/] - No changes made")
+ return
+
+ setup_vscode()
+
+ print("\n[green]Setup complete![/]")
+ print("\nFor more information, see:
contributing-docs/20_debugging_airflow_components.rst")
+
+
+if __name__ == "__main__":
+ main()
