This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new bfb121a2ee5 Split task execution architecture docs: concise overview +
dev guide (#70657)
bfb121a2ee5 is described below
commit bfb121a2ee5e3b25e670d07b6b192c17eab67eac
Author: Jason(Zhe-You) Liu <[email protected]>
AuthorDate: Thu Aug 20 13:20:57 2026 +0800
Split task execution architecture docs: concise overview + dev guide
(#70657)
* Move task execution architecture docs to contributing-docs
The "Task execution architecture" section added in #69750 documents
worker internals (the Supervisor, the Coordinator layer, and the language
subprocess protocols). That is developer-facing material, not something
end users need in the core-concepts overview. Relocate the whole section
and its diagrams to a new dev-facing page under contributing-docs, next
to the new-language-SDK guide, and drop it from the user-facing overview.
Follow-up to review feedback on #70657.
* Add task execution architecture section to Architecture Overview
The overview explained how components are deployed but not what happens
inside a worker when a task actually runs. This adds a concise section and
diagram distinguishing the supervisor process, the CoordinatorManager that
routes a task to a coordinator by its queue, and the language SDK subprocess
launched for non-Python tasks, so readers can reason about the process
boundaries. The full developer-level detail remains in contributing-docs.
* Simplify worker task execution diagram in Architecture Overview
Deployment Managers reading the Architecture Overview need to know what
runs on a worker while tasks execute, so they can size workers for the
memory and start-up cost each concurrent task instance adds. Coordinator
routing, the msgpack wire protocol and per-runtime internals do not serve
that question and already live in the developer guide.
The previous wording also implied Python tasks run inside the supervisor
with no separate process, which is not what happens: every task instance
gets its own subprocess whatever language it is written in.
* Highlight task execution lifecycle documentation
---
.pre-commit-config.yaml | 1 +
airflow-core/docs/core-concepts/overview.rst | 92 +++---------
.../diagram_java_sdk_execution_architecture.md5sum | 1 -
.../diagram_java_sdk_execution_architecture.png | Bin 176205 -> 0 bytes
.../img/diagram_java_sdk_execution_sequence.md5sum | 1 -
.../img/diagram_java_sdk_execution_sequence.png | Bin 189307 -> 0 bytes
...diagram_native_language_sdk_architecture.md5sum | 1 -
.../img/diagram_task_execution_architecture.md5sum | 1 +
.../img/diagram_task_execution_architecture.png | Bin 0 -> 47036 bytes
.../img/diagram_task_execution_architecture.py | 154 +++++++++++++++++++++
contributing-docs/30_new_language_sdk.rst | 3 +
.../31_task_execution_architecture.rst | 116 ++++++++++++++++
contributing-docs/README.rst | 8 ++
...agram_coordinator_execution_architecture.md5sum | 1 +
.../diagram_coordinator_execution_architecture.png | Bin 0 -> 185035 bytes
.../diagram_coordinator_execution_architecture.py | 110 ++++++++-------
.../diagram_coordinator_execution_sequence.md5sum | 1 +
.../diagram_coordinator_execution_sequence.png | Bin 0 -> 219883 bytes
.../diagram_coordinator_execution_sequence.py | 72 +++++-----
.../diagram_go_edge_worker_architecture.md5sum | 1 +
.../images/diagram_go_edge_worker_architecture.png | Bin
.../images/diagram_go_edge_worker_architecture.py | 17 ++-
.../diagram_task_sdk_execution_architecture.md5sum | 0
.../diagram_task_sdk_execution_architecture.png | Bin
.../diagram_task_sdk_execution_architecture.py | 0
.../diagram_task_sdk_execution_sequence.md5sum | 0
.../diagram_task_sdk_execution_sequence.png | Bin
.../images}/diagram_task_sdk_execution_sequence.py | 0
28 files changed, 412 insertions(+), 168 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 4dfa0aa7fc2..c00e0fb7e0f 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -664,6 +664,7 @@ repos:
files: >
(?x)
^airflow-core/docs/.*/diagram_[^/]*\.py$|
+ ^contributing-docs/images/diagram_[^/]*\.py$|
^docs/images/.*\.py$|
^airflow-ctl/docs/images/diagrams/.*\.py$
pass_filenames: true
diff --git a/airflow-core/docs/core-concepts/overview.rst
b/airflow-core/docs/core-concepts/overview.rst
index 1f2b1ad8a11..24741ffdfea 100644
--- a/airflow-core/docs/core-concepts/overview.rst
+++ b/airflow-core/docs/core-concepts/overview.rst
@@ -192,90 +192,30 @@ code is never executed in the context of the *scheduler*.
bundle version when dispatching each task. If needed, the cadence of sync
and scan
of the *Dag bundle* can be configured.
-Task execution architecture
----------------------------
-
-The diagrams above show how Airflow's components are *deployed*. The diagrams
below instead show what happens
-*inside a worker when a task actually runs* — how the Task SDK, the Supervisor
and Coordinator processes, and
-the language runtimes work together: which processes are involved, and the
classes and protocols they use to
-communicate.
-
-.. _overview-task-sdk-execution-architecture:
-
-Python Task SDK execution
-.........................
-
-When a *worker* actually runs a task, it does not run the user's code
directly. Instead it starts a
-lightweight **Supervisor** that runs in its own **native operating-system
process** and
-*forks* a second native process in which the **Task SDK** runtime
(``task_runner``) executes the user code.
-The two processes talk over a socket, and the Supervisor is the only side that
ever holds the short-lived
-task JWT or talks to the *Execution API* — the user's code never sees the
token and never touches the
-database.
-
-The same runtime can also run *in-process* (a single Python process, no fork,
no sockets, no HTTP) for
-``dag.test()`` and local runs. The diagram below contrasts the two paths and
marks where each Python process
-lives:
-
-.. image:: ../img/diagram_task_sdk_execution_architecture.png
-
-The message flow of a supervised run — startup, running the user code, proxied
Connection/Variable/XCom
-lookups, heartbeats, and reporting the final state — is shown below as a
sequence diagram, with each process
-on its own lifeline. The **Supervisor** sits in the middle, so the Task ↔
Supervisor request/response
-round-trip (the task asks for a Connection/Variable/XCom and gets the answer
back) reads as arrows going back
-and forth between neighboring lifelines. Each arrow is numbered, colored by
its sender, and labeled with the
-message class or protocol used:
-
-.. image:: ../img/diagram_task_sdk_execution_sequence.png
+.. _overview-task-execution-architecture:
-.. _overview-non-python-language-sdks:
-
-Non-Python language SDKs (Go and Java)
-......................................
-
-The Task Execution Interface (TEI) introduced in AIP-72 is language-agnostic,
so a task can also be written in
-a **compiled, non-Python language**. A Python Dag still declares the task with
``@task.stub(queue=...)`` (so
-Python and non-Python tasks can be mixed in one Dag), but the actual work is
delegated to the matching runtime.
-There are currently **two different integration styles** — the Go SDK runs a
standalone worker, while the Java
-SDK plugs into the existing Python Supervisor.
-
-.. _overview-go-sdk-architecture:
-
-**Go SDK — standalone edge worker.** The `Go Task SDK
-<https://github.com/apache/airflow/blob/main/go-sdk/README.md>`_ has **no
Python Supervisor and no msgpack
-stdin socket**. A long-running, compiled **edge worker**
(``airflow-go-edge-worker``) *pulls* work from the
-**Edge Executor API**, launches the user's compiled Dag bundle as a
**go-plugin (gRPC) subprocess**, and
-invokes the task over gRPC. The task then uses the **native TEI client** to
reach the **Execution API**
-directly over HTTPS — so, unlike the Python task, it holds the task JWT itself:
-
-.. image:: ../img/diagram_native_language_sdk_architecture.png
+Task execution architecture
+----------------------------
-.. _overview-java-sdk-architecture:
+The diagrams above show how Airflow's components are *deployed*.
+This section covers what you should expect to be running on a *worker* while
tasks execute, so you can size workers for the memory and start-up cost that
each concurrent task instance adds.
-**Java (JVM) SDK — Coordinator plugged into the Supervisor.** The `Java Task
SDK
-<https://github.com/apache/airflow/blob/main/java-sdk/README.md>`_ takes the
opposite approach: it *reuses* the
-existing Python Supervisor through a new **Coordinator** layer.
``CoordinatorManager`` resolves the task's
-``queue`` to a ``BaseCoordinator`` — ``JavaCoordinator`` for the ``java``
queue, or the built-in
-``_PythonCoordinator`` otherwise. ``JavaCoordinator`` opens two loopback-TCP
servers, spawns a **JVM bundle
-process** with ``subprocess.Popen``, and drives it with
``_JavaActivitySubprocess`` (a subclass of the shared
-``ActivitySubprocess``). The JVM connects *back* over TCP and speaks the
**same msgpack protocol** as a Python
-task, so the Python side heartbeats, manages state, and **proxies every
Execution-API call** — meaning the JVM
-task, like a Python task, never holds the task JWT itself:
+A *worker* never runs task code in its own process.
+For every task instance it starts a **new subprocess**, supervises it, and
tears it down once the task instance finishes.
+That is true whatever language the task is written in — only the kind of
subprocess differs:
-.. image:: ../img/diagram_java_sdk_execution_architecture.png
+* **Python** — a forked Python interpreter.
+* **Java** — a brand-new JVM instance.
+* **Go** — a brand-new process of the compiled task binary.
-The end-to-end workflow of a Java task — from ``@task.stub`` through the
coordinator, the JVM subprocess, the
-proxied Connection/Variable/XCom lookups, and reporting the final state — is
shown below as a sequence diagram.
-As above, the **Supervisor** is the central lifeline, so the JVM ↔ Supervisor
round-trip over loopback TCP is
-drawn as arrows going back and forth to its neighbours:
+.. image:: ../img/diagram_task_execution_architecture.png
-.. image:: ../img/diagram_java_sdk_execution_sequence.png
+Nothing is pooled or reused between task instances, so *N* concurrent task
instances cost *N* subprocesses plus anything the task code itself spawns.
+Sizing a worker therefore means budgeting for the peak number of task
instances it runs at once, not for the worker process alone.
-.. note::
+The worker process is also the only side that holds the task's short-lived
credentials and talks to the *Execution API*, so the subprocess running the
task code never reaches the metadata database directly, in any language.
- Both the Go and Java SDKs are **experimental** and under active
development. See the `Go Task SDK
- documentation
<https://github.com/apache/airflow/blob/main/go-sdk/README.md>`_ and the `Java
Task SDK
- documentation
<https://github.com/apache/airflow/blob/main/java-sdk/README.md>`_ for current
status,
- quick-starts, and known limitations.
+For how the worker drives each runtime internally — the wire protocol and
sequence diagrams — see the `task execution architecture developer guide
<https://github.com/apache/airflow/blob/main/contributing-docs/31_task_execution_architecture.rst>`_.
.. _overview:workloads:
diff --git
a/airflow-core/docs/img/diagram_java_sdk_execution_architecture.md5sum
b/airflow-core/docs/img/diagram_java_sdk_execution_architecture.md5sum
deleted file mode 100644
index 377507bd884..00000000000
--- a/airflow-core/docs/img/diagram_java_sdk_execution_architecture.md5sum
+++ /dev/null
@@ -1 +0,0 @@
-061e71105ed4ce16e740a942073b7ace
diff --git a/airflow-core/docs/img/diagram_java_sdk_execution_architecture.png
b/airflow-core/docs/img/diagram_java_sdk_execution_architecture.png
deleted file mode 100644
index 8fa1e5e6255..00000000000
Binary files
a/airflow-core/docs/img/diagram_java_sdk_execution_architecture.png and
/dev/null differ
diff --git a/airflow-core/docs/img/diagram_java_sdk_execution_sequence.md5sum
b/airflow-core/docs/img/diagram_java_sdk_execution_sequence.md5sum
deleted file mode 100644
index 590d397530f..00000000000
--- a/airflow-core/docs/img/diagram_java_sdk_execution_sequence.md5sum
+++ /dev/null
@@ -1 +0,0 @@
-5c927fe1f455ae4fc5bf77ebf7fc7ae7
diff --git a/airflow-core/docs/img/diagram_java_sdk_execution_sequence.png
b/airflow-core/docs/img/diagram_java_sdk_execution_sequence.png
deleted file mode 100644
index 71cd2ebbc4a..00000000000
Binary files a/airflow-core/docs/img/diagram_java_sdk_execution_sequence.png
and /dev/null differ
diff --git
a/airflow-core/docs/img/diagram_native_language_sdk_architecture.md5sum
b/airflow-core/docs/img/diagram_native_language_sdk_architecture.md5sum
deleted file mode 100644
index 8149286f363..00000000000
--- a/airflow-core/docs/img/diagram_native_language_sdk_architecture.md5sum
+++ /dev/null
@@ -1 +0,0 @@
-d7f0c11f0b175a82b2bded558ebfa0f1
diff --git a/airflow-core/docs/img/diagram_task_execution_architecture.md5sum
b/airflow-core/docs/img/diagram_task_execution_architecture.md5sum
new file mode 100644
index 00000000000..09c8fed4083
--- /dev/null
+++ b/airflow-core/docs/img/diagram_task_execution_architecture.md5sum
@@ -0,0 +1 @@
+8f1b611206d52ba7d534f4fffaffe895
diff --git a/airflow-core/docs/img/diagram_task_execution_architecture.png
b/airflow-core/docs/img/diagram_task_execution_architecture.png
new file mode 100644
index 00000000000..2c4f357f31b
Binary files /dev/null and
b/airflow-core/docs/img/diagram_task_execution_architecture.png differ
diff --git a/airflow-core/docs/img/diagram_task_execution_architecture.py
b/airflow-core/docs/img/diagram_task_execution_architecture.py
new file mode 100644
index 00000000000..1ff2ce8c86b
--- /dev/null
+++ b/airflow-core/docs/img/diagram_task_execution_architecture.py
@@ -0,0 +1,154 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "graphviz>=0.20.1",
+# ]
+# ///
+"""
+Worker process model for the *Architecture Overview* page.
+
+Answers the one question a Deployment Manager needs in order to size a worker:
+*what runs, and for how long, while a task instance executes?* Every task
+instance gets its own subprocess whatever language it is written in, and that
+subprocess lives exactly as long as the task instance — for Java and Go it is a
+fresh JVM / fresh binary process each time, not a pooled runtime.
+
+The worker process's own lifetime is deliberately left unstated: it is pooled
and
+reused under the Local and Celery executors, but one-shot per task instance
under
+the Kubernetes executor. Only the task subprocess behaves the same way
everywhere.
+
+Deliberately omits how the worker and the subprocess talk to each other
+(coordinators, the msgpack comm frame, the Execution API); that detail is
+developer-facing and lives in
``contributing-docs/31_task_execution_architecture.rst``.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import graphviz
+from rich.console import Console
+
+MY_DIR = Path(__file__).parent
+MY_FILENAME = Path(__file__).with_suffix("").name
+
+console = Console(width=400, color_system="standard")
+
+# (fill, border) per role — consistent with the detailed Task SDK diagrams.
+WORKER = ("#e3f2fd", "#1565c0") # the long-running worker process (blue)
+PY = ("#e8f5e9", "#2e7d32") # Python task subprocess (green)
+LANG = ("#fbe9e7", "#d84315") # JVM / native binary task subprocess (deep
orange)
+
+# One box per task instance: (node id, heading, what the subprocess actually
is, theme).
+TASK_INSTANCES = (
+ ("py", "Task instance · Python", "a forked Python interpreter", PY),
+ ("java", "Task instance · Java", "a brand-new JVM instance", LANG),
+ ("go", "Task instance · Go", "a brand-new process of the compiled
binary", LANG),
+)
+
+
+def _label(title: str, sub: str | None = None) -> str:
+ html = f"<<b>{title}</b>"
+ if sub:
+ html += f'<br/><font point-size="11" color="#37474f">{sub}</font>'
+ return html + ">"
+
+
+def _node(g, node_id: str, title: str, sub: str, *, shape: str, theme:
tuple[str, str]) -> None:
+ fill, border = theme
+ style = "filled" if shape == "box3d" else "rounded,filled"
+ g.node(
+ node_id,
+ label=_label(title, sub),
+ shape=shape,
+ style=style,
+ fillcolor=fill,
+ color=border,
+ penwidth="2",
+ margin="0.20,0.14",
+ )
+
+
+def generate_task_execution_architecture_diagram():
+ image_file = MY_DIR / f"{MY_FILENAME}.png"
+ console.print(f"[bright_blue]Generating architecture image {image_file}")
+
+ g = graphviz.Digraph("task_execution_architecture")
+ g.attr(
+ rankdir="TB",
+ splines="spline",
+ nodesep="0.7",
+ ranksep="1.1",
+ pad="0.5",
+ bgcolor="white",
+ fontname="Helvetica",
+ newrank="true",
+ )
+ g.attr("node", fontname="Helvetica", fontsize="13", fontcolor="#102027")
+ g.attr("edge", fontname="Helvetica", fontsize="11", penwidth="1.8",
color="#546e7a")
+
+ with g.subgraph(name="cluster_worker") as worker:
+ worker.attr(
+ label="Worker",
+ labelloc="t",
+ style="rounded,filled",
+ fillcolor="#fafafa",
+ color="#607d8b",
+ penwidth="1.8",
+ fontsize="19",
+ fontname="Helvetica-Bold",
+ margin="24",
+ )
+ _node(
+ worker,
+ "worker",
+ "Worker process",
+ "supervises the task instance · holds its credentials ·
runs NO user code",
+ shape="box",
+ theme=WORKER,
+ )
+
+ for node_id, heading, subprocess_kind, theme in TASK_INSTANCES:
+ _node(
+ worker,
+ node_id,
+ heading,
+ f"{subprocess_kind} · runs USER CODE",
+ shape="box3d",
+ theme=theme,
+ )
+
+ for index, (node_id, _, _, theme) in enumerate(TASK_INSTANCES):
+ g.edge(
+ "worker",
+ node_id,
+ color=theme[1],
+ arrowhead="vee",
+ # Label the middle edge only — all three edges mean the same
thing, and repeating
+ # the text three times crowds the fan-out.
+ label="starts a new subprocess\nper task instance" if index == 1
else "",
+ )
+
+ g.render(outfile=str(image_file), format="png", cleanup=True)
+ console.print(f"[green]Generated architecture image {image_file}")
+
+
+if __name__ == "__main__":
+ generate_task_execution_architecture_diagram()
diff --git a/contributing-docs/30_new_language_sdk.rst
b/contributing-docs/30_new_language_sdk.rst
index 37b20b33dbd..69ebccc23ba 100644
--- a/contributing-docs/30_new_language_sdk.rst
+++ b/contributing-docs/30_new_language_sdk.rst
@@ -38,6 +38,9 @@ start the foreign runtime and *how* to communicate with it,
and the language SDK
implements the other end of whatever protocol the coordinator chooses. There is
no single mandated communication mechanism.
+For a diagram-driven overview of how a task runs across the Python,
Coordinator,
+and edge-worker paths, see `Task execution architecture
<31_task_execution_architecture.rst>`__.
+
.. contents:: Table of Contents
:depth: 2
:local:
diff --git a/contributing-docs/31_task_execution_architecture.rst
b/contributing-docs/31_task_execution_architecture.rst
new file mode 100644
index 00000000000..6c9419accca
--- /dev/null
+++ b/contributing-docs/31_task_execution_architecture.rst
@@ -0,0 +1,116 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Task execution architecture
+===========================
+
+This page is developer-facing background on what happens *inside a worker when
a task actually runs* — how the
+Task SDK, the Supervisor and Coordinator processes, and the language runtimes
work together: which processes
+are involved, and the classes and protocols they use to communicate. It
complements
+`Adding a new language SDK <30_new_language_sdk.rst>`__, which covers how to
implement the coordinator side of
+a new runtime.
+
+End users do not need any of this: they only configure a queue and write
tasks. The user-facing guides live
+under `Non-Python Task SDKs
<../airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst>`__.
+
+.. contents:: Table of Contents
+ :local:
+ :depth: 2
+
+Python Task SDK execution
+-------------------------
+
+When a *worker* actually runs a task, it does not run the user's code
directly. Instead it starts a
+lightweight **Supervisor** that runs in its own **native operating-system
process** and
+*forks* a second native process in which the **Task SDK** runtime
(``task_runner``) executes the user code.
+The two processes talk over a socket, and the Supervisor is the only side that
ever holds the short-lived
+task JWT or talks to the *Execution API* — the user's code never sees the
token and never touches the
+database.
+
+The same runtime can also run *in-process* (a single Python process, no fork,
no sockets, no HTTP) for
+``dag.test()`` and local runs. The diagram below contrasts the two paths and
marks where each Python process
+lives:
+
+.. image:: images/diagram_task_sdk_execution_architecture.png
+
+The message flow of a supervised run — startup, running the user code, proxied
Connection/Variable/XCom
+lookups, heartbeats, and reporting the final state — is shown below as a
sequence diagram, with each process
+on its own lifeline. The **Supervisor** sits in the middle, so the Task ↔
Supervisor request/response
+round-trip (the task asks for a Connection/Variable/XCom and gets the answer
back) reads as arrows going back
+and forth between neighboring lifelines. Each arrow is numbered, colored by
its sender, and labeled with the
+message class or protocol used:
+
+.. image:: images/diagram_task_sdk_execution_sequence.png
+
+Non-Python language SDKs (Go and Java)
+--------------------------------------
+
+The Task Execution Interface (TEI) introduced in AIP-72 is language-agnostic,
so a task can also be written in
+a **compiled, non-Python language**. A Python Dag still declares the task with
``@task.stub(queue=...)`` (so
+Python and non-Python tasks can be mixed in one Dag), but the actual work is
delegated to the matching runtime.
+The **first-class** integration is the **Coordinator** layer — the Python
Supervisor drives the language runtime
+as a subprocess and proxies every Execution-API call for it — and it is the
shared direction for the Java, Go,
+and upcoming language SDKs. The Go SDK additionally offers a second,
standalone **edge-worker** mode.
+
+**Coordinator (Java and Go) — recommended.** Both the
+`Java SDK
<../airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst>`__ and
the
+`Go SDK
<../airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst>`__ *reuse*
the existing Python
+Supervisor through the **Coordinator** layer. ``CoordinatorManager`` resolves
the task's ``queue`` to a
+``BaseCoordinator`` — a ``SubprocessCoordinator`` (``JavaCoordinator`` for the
``java`` queue,
+``ExecutableCoordinator`` for the Go bundle queue, e.g. ``golang``), or the
built-in ``_PythonCoordinator``
+otherwise. The coordinator opens two loopback-TCP servers, spawns the language
**bundle subprocess**
+(``java -classpath ... <main class>`` for Java, or the self-contained packed
bundle binary for Go) with
+``--comm`` / ``--logs`` appended, and drives it with
``_PopenActivitySubprocess`` (a subclass of the shared
+``ActivitySubprocess``). The subprocess connects *back* over TCP and speaks
the **same msgpack protocol** as a
+Python task, so the Python side heartbeats, manages state, and **proxies every
Execution-API call** — meaning
+the language task, like a Python task, never holds the task JWT itself.
Because the mature Python Supervisor
+handles the Airflow-facing concerns, this mode inherits its capabilities —
remote task logs (S3/GCS, etc.), the
+full range of task states, and alternate XCom backends:
+
+.. image:: images/diagram_coordinator_execution_architecture.png
+
+The end-to-end workflow of a coordinator task — from ``@task.stub`` through
the coordinator, the language
+subprocess, the proxied Connection/Variable/XCom lookups, and reporting the
final state — is shown below as a
+sequence diagram. The **Supervisor** is the central lifeline, so the
subprocess ↔ Supervisor round-trip over
+loopback TCP is drawn as arrows going back and forth to its neighbours:
+
+.. image:: images/diagram_coordinator_execution_sequence.png
+
+**Go SDK — standalone edge worker (alternative).** The Go SDK can also run
with **no Python in the data
+path**, but this mode is still missing features the Supervisor provides —
non-default task states, remote task
+logs, and alternate XCom backends (see the Go SDK
+`limitations
<../airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst#limitations>`__)
— so the
+Coordinator mode above stays the recommended path. When it is used, a
long-running, compiled **edge worker**
+(``airflow-go-edge-worker``) *pulls* work from the **Edge Executor API**,
launches the user's compiled Dag
+bundle as a **go-plugin (gRPC) subprocess**, and invokes the task over gRPC.
The task then uses the **native
+TEI client** to reach the **Execution API** directly over HTTPS — so, unlike
the coordinator path, it holds the
+task JWT itself:
+
+.. image:: images/diagram_go_edge_worker_architecture.png
+
+.. note::
+
+ Both the Go and Java SDKs are **experimental** and under active
development. See the
+ `Non-Python Task SDKs guide
<../airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst>`__ —
+ the `Go SDK
<../airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst>`__ and
+ `Java SDK
<../airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst>`__ pages
— for current
+ status, quick-starts, and known limitations.
+
+------
+
+To implement the coordinator side of a new runtime, continue to
+`Adding a new language SDK <30_new_language_sdk.rst>`__.
diff --git a/contributing-docs/README.rst b/contributing-docs/README.rst
index f27754edf03..823ed47d545 100644
--- a/contributing-docs/README.rst
+++ b/contributing-docs/README.rst
@@ -160,3 +160,11 @@ Maintainer Tools
* `CLI Implementation Guide <27_cli_implementation_guide.rst>`__ describes
where to implement new
CLI features following AIP-94: remote commands go to ``airflowctl``,
admin/deployment commands
stay in the ``airflow`` CLI.
+
+
+Task Execution Lifecycle
+........................
+
+* `Task execution architecture <31_task_execution_architecture.rst>`__
explains what happens inside a
+ worker when a task runs: the Python Supervisor and task runner, the
Coordinator layer shared by the Java
+ and Go SDKs, and the standalone Go edge worker.
diff --git
a/contributing-docs/images/diagram_coordinator_execution_architecture.md5sum
b/contributing-docs/images/diagram_coordinator_execution_architecture.md5sum
new file mode 100644
index 00000000000..c19d1feaf7f
--- /dev/null
+++ b/contributing-docs/images/diagram_coordinator_execution_architecture.md5sum
@@ -0,0 +1 @@
+274a2f3f7e4d8ef8b7807e3f5234eabd
diff --git
a/contributing-docs/images/diagram_coordinator_execution_architecture.png
b/contributing-docs/images/diagram_coordinator_execution_architecture.png
new file mode 100644
index 00000000000..22668ad8d1e
Binary files /dev/null and
b/contributing-docs/images/diagram_coordinator_execution_architecture.png differ
diff --git a/airflow-core/docs/img/diagram_java_sdk_execution_architecture.py
b/contributing-docs/images/diagram_coordinator_execution_architecture.py
similarity index 67%
rename from airflow-core/docs/img/diagram_java_sdk_execution_architecture.py
rename to contributing-docs/images/diagram_coordinator_execution_architecture.py
index 25be4210622..aa7a158f6d3 100644
--- a/airflow-core/docs/img/diagram_java_sdk_execution_architecture.py
+++ b/contributing-docs/images/diagram_coordinator_execution_architecture.py
@@ -22,19 +22,27 @@
# ]
# ///
"""
-Architecture diagram for the Java (JVM) Task SDK.
+Architecture diagram for the Coordinator execution path (Java and Go language
SDKs).
-Unlike the Go SDK (a standalone edge worker), the Java SDK plugs into the
*same*
-Python Supervisor via a new **Coordinator** layer:
+Both plug into the *same* Python Supervisor via the **Coordinator** layer,
rather
+than running a standalone worker:
-* ``CoordinatorManager`` resolves the task's ``queue`` to a ``BaseCoordinator``
- (``JavaCoordinator`` for the ``java`` queue, ``_PythonCoordinator``
otherwise);
-* ``JavaCoordinator.execute_task()`` opens two loopback-TCP servers, spawns a
JVM
- bundle process with ``subprocess.Popen``, and drives it with
- ``_JavaActivitySubprocess`` — a subclass of the shared
``ActivitySubprocess``;
-* the JVM process connects *back* over TCP and speaks the same msgpack
protocol as
- a Python task, so the Python side heartbeats, proxies every Execution-API
call,
- and manages state. The JVM task therefore **never holds the task JWT**.
+* ``CoordinatorManager.for_queue()`` resolves the task's ``queue`` to a
+ ``BaseCoordinator`` via ``[sdk] queue_to_coordinator`` / ``[sdk]
coordinators``
+ (``_PythonCoordinator`` for unmapped queues);
+* the resolved ``SubprocessCoordinator`` (``JavaCoordinator`` for Java,
+ ``ExecutableCoordinator`` for Go) binds two loopback-TCP servers, builds the
+ per-language launch command, spawns the bundle subprocess with ``--comm`` /
+ ``--logs`` appended, and drives it through ``_PopenActivitySubprocess`` — a
+ subclass of the shared ``ActivitySubprocess``;
+* the language subprocess connects *back* over TCP and speaks the same msgpack
+ protocol as a Python task, so the Python side heartbeats, proxies every
+ Execution-API call, and manages state. The language task therefore **never
+ holds the task JWT**.
+
+The launch command is the only per-language difference: Java runs
+``java -classpath ... <main class>``; Go execs the self-contained packed bundle
+binary directly.
Rendered with graphviz directly so labels sit inside sized shapes: 3-D box =
native OS process, rounded box = an object inside a process, component = a
server
@@ -56,7 +64,7 @@ console = Console(width=400, color_system="standard")
# (fill, border) per role — consistent with the other Task SDK diagrams.
COORD = ("#ede7f6", "#5e35b1") # Coordinator layer (Python, deep purple)
SUP = ("#e3f2fd", "#1565c0") # Supervisor / ActivitySubprocess + Client (blue)
-JVM = ("#fbe9e7", "#d84315") # JVM SDK runtime (deep orange)
+LANG = ("#fbe9e7", "#d84315") # language SDK runtime — JVM or compiled binary
(deep orange)
USER = ("#e8f5e9", "#2e7d32") # user task code (green)
API = ("#fdecea", "#c62828") # Execution API (red)
NEUTRAL = ("#eceff1", "#455a64") # database
@@ -85,11 +93,11 @@ def _node(g, node_id: str, title: str, sub: str, *, shape:
str, theme: tuple[str
)
-def generate_java_sdk_execution_architecture_diagram():
+def generate_coordinator_execution_architecture_diagram():
image_file = MY_DIR / f"{MY_FILENAME}.png"
console.print(f"[bright_blue]Generating architecture image {image_file}")
- g = graphviz.Digraph("java_sdk_execution_architecture")
+ g = graphviz.Digraph("coordinator_execution_architecture")
g.attr(
rankdir="TB",
splines="spline",
@@ -142,16 +150,18 @@ def generate_java_sdk_execution_architecture_diagram():
)
_node(
sup,
- "java_coord",
- "JavaCoordinator (BaseCoordinator)",
- "execute_task(client): open two loopback-TCP<br/>servers,
subprocess.Popen(java -jar bundle)",
+ "coord",
+ "SubprocessCoordinator",
+ "JavaCoordinator (java) · ExecutableCoordinator (golang)<br/>"
+ "bind two loopback-TCP servers · spawn bundle subprocess<br/>"
+ "with --comm / --logs appended",
shape="box",
theme=COORD,
)
_node(
sup,
"supervisor",
- "_JavaActivitySubprocess",
+ "_PopenActivitySubprocess",
"subclass of the shared ActivitySubprocess<br/>heartbeat ·
proxy every API call · manage state",
shape="box3d",
theme=SUP,
@@ -164,60 +174,58 @@ def generate_java_sdk_execution_architecture_diagram():
shape="box",
theme=SUP,
)
- sup.edge(
- "coord_mgr", "java_coord", style="dotted", color=COORD[1],
arrowhead="vee", label="picks"
- )
- sup.edge(
- "java_coord", "supervisor", style="dotted", color=SUP[1],
arrowhead="vee", label="drives"
- )
+ sup.edge("coord_mgr", "coord", style="dotted", color=COORD[1],
arrowhead="vee", label="picks")
+ sup.edge("coord", "supervisor", style="dotted", color=SUP[1],
arrowhead="vee", label="drives")
sup.edge("supervisor", "client", style="dotted", color=SUP[1],
arrowhead="none")
- with host.subgraph(name="cluster_jvm") as jvm:
- jvm.attr(
- label="JVM bundle subprocess · native OS process (JVM) ·
runs USER CODE",
+ with host.subgraph(name="cluster_lang") as lang:
+ lang.attr(
+ label="Language bundle subprocess · native OS process (JVM
or compiled binary) · runs USER CODE",
labelloc="t",
style="rounded,filled",
fillcolor="#fdefe9",
- color=JVM[1],
+ color=LANG[1],
penwidth="1.5",
fontsize="13",
fontname="Helvetica-Bold",
margin="14",
)
_node(
- jvm,
+ lang,
"server",
- "Server.serve(bundle)",
- "bundle JAR entry point (Main-Class)<br/>Kotlin runtime ·
connects back over TCP",
+ "bundle entry point",
+ "Java: java -classpath … main class (JVM)<br/>"
+ "Go: exec the packed bundle binary (compiled)<br/>"
+ "connects back over TCP",
shape="box3d",
- theme=JVM,
+ theme=LANG,
)
_node(
- jvm,
+ lang,
"task",
- "Task.execute(Context, Client)",
- "your Java / Kotlin task [user code]<br/>Context = static run
data · Client = API accessors",
+ "task function [user code]",
+ "your Java / Kotlin or Go task<br/>Context = run data · Client
= API accessors",
shape="box3d",
theme=USER,
)
_node(
- jvm,
+ lang,
"comm",
"CoordinatorComm / Frame",
- "msgpack framing (Kotlin)",
+ "length-prefixed msgpack-over-IPC framing",
shape="box",
- theme=JVM,
+ theme=LANG,
)
- jvm.edge("server", "task", style="dotted", color=JVM[1],
arrowhead="vee", label="invokes")
- jvm.edge("task", "comm", style="dotted", color=JVM[1],
arrowhead="none")
+ lang.edge("server", "task", style="dotted", color=LANG[1],
arrowhead="vee", label="invokes")
+ lang.edge("task", "comm", style="dotted", color=LANG[1],
arrowhead="none")
- # Loopback-TCP comms between the two native OS processes (JVM connects
back).
+ # Loopback-TCP comms between the two native OS processes (the
subprocess connects back).
g.edge(
"comm",
"supervisor",
- label="msgpack frames over loopback TCP (127.0.0.1)\nJVM connects
back · ToSupervisor / ToTask",
- color=JVM[1],
- fontcolor=JVM[1],
+ label="msgpack frames over loopback TCP (127.0.0.1)\nsubprocess
connects back · ToSupervisor / ToTask",
+ color=LANG[1],
+ fontcolor=LANG[1],
dir="both",
penwidth="2.2",
)
@@ -225,7 +233,7 @@ def generate_java_sdk_execution_architecture_diagram():
"comm",
"supervisor",
label="structured logs\n(second TCP channel)",
- color=JVM[1],
+ color=LANG[1],
fontcolor="#a1674f",
style="dashed",
)
@@ -259,7 +267,7 @@ def generate_java_sdk_execution_architecture_diagram():
g.edge(
"client",
"execution_api",
- label="HTTPS + task JWT\n(proxied for the JVM task)",
+ label="HTTPS + task JWT\n(proxied for the language task)",
color=API[1],
fontcolor=API[1],
penwidth="2.2",
@@ -270,10 +278,10 @@ def generate_java_sdk_execution_architecture_diagram():
g,
"note",
"How it differs",
- "vs Python task: same Supervisor, but a JVM subprocess over loopback
TCP<br/>"
- "instead of a forked Python process over a UNIX socketpair<br/>"
- "vs Go SDK: the JVM task does <b>not</b> hold the JWT and does
<b>not</b> call the<br/>"
- "Execution API directly — the Python Supervisor proxies every call",
+ "vs Python task: same Supervisor, but a separate language subprocess
over<br/>"
+ "loopback TCP instead of a forked Python process over a UNIX
socketpair<br/>"
+ "vs the Go edge worker: the coordinator task does <b>not</b> hold the
JWT and does<br/>"
+ "<b>not</b> call the Execution API directly — the Python Supervisor
proxies every call",
shape="note",
theme=NOTE,
)
@@ -284,4 +292,4 @@ def generate_java_sdk_execution_architecture_diagram():
if __name__ == "__main__":
- generate_java_sdk_execution_architecture_diagram()
+ generate_coordinator_execution_architecture_diagram()
diff --git
a/contributing-docs/images/diagram_coordinator_execution_sequence.md5sum
b/contributing-docs/images/diagram_coordinator_execution_sequence.md5sum
new file mode 100644
index 00000000000..727ecf599e0
--- /dev/null
+++ b/contributing-docs/images/diagram_coordinator_execution_sequence.md5sum
@@ -0,0 +1 @@
+174d92296bae3b1f391942a090685977
diff --git
a/contributing-docs/images/diagram_coordinator_execution_sequence.png
b/contributing-docs/images/diagram_coordinator_execution_sequence.png
new file mode 100644
index 00000000000..6ae54d1f7fe
Binary files /dev/null and
b/contributing-docs/images/diagram_coordinator_execution_sequence.png differ
diff --git a/airflow-core/docs/img/diagram_java_sdk_execution_sequence.py
b/contributing-docs/images/diagram_coordinator_execution_sequence.py
similarity index 80%
rename from airflow-core/docs/img/diagram_java_sdk_execution_sequence.py
rename to contributing-docs/images/diagram_coordinator_execution_sequence.py
index 3c9d4ae9658..dde9178b01f 100644
--- a/airflow-core/docs/img/diagram_java_sdk_execution_sequence.py
+++ b/contributing-docs/images/diagram_coordinator_execution_sequence.py
@@ -22,22 +22,26 @@
# ]
# ///
"""
-UML-style sequence diagram for a Java (JVM) task run.
+UML-style sequence diagram for a Coordinator task run (Java and Go language
SDKs).
Each participant gets its own vertical lifeline; messages are horizontal arrows
between lifelines, read top to bottom. The **Supervisor** (Python) sits in the
-middle so the JVM <-> Supervisor round-trip (the JVM asks for a
+middle so the subprocess <-> Supervisor round-trip (the language task asks for
a
Connection/Variable/XCom over loopback TCP and gets the answer back) and the
Supervisor <-> Execution API round-trip are both drawn as adjacent request/
-response pairs. The JVM task never talks to the Execution API — the Supervisor
-proxies every call, so the JVM never holds the task JWT.
+response pairs. The language task never talks to the Execution API — the
+Supervisor proxies every call, so the task never holds the task JWT.
+
+Only the spawn step differs per language: Java runs ``java -classpath ... <main
+class>``; Go execs the self-contained packed bundle binary directly. Everything
+after the TCP connect-back is identical.
Arrows are colored by sender:
* teal — Scheduler
-* purple — Coordinator layer (CoordinatorManager / JavaCoordinator)
-* blue — Supervisor (_JavaActivitySubprocess) → JVM / Execution API
-* orange — JVM SDK runtime / user code → Supervisor (over loopback TCP)
+* purple — Coordinator layer (CoordinatorManager / SubprocessCoordinator)
+* blue — Supervisor (_PopenActivitySubprocess) → subprocess / Execution API
+* orange — language SDK runtime / user code → Supervisor (over loopback TCP)
* red — Execution API → Supervisor (responses)
Graphviz has no native sequence-diagram shape, so lifelines are drawn as dashed
@@ -62,10 +66,10 @@ console = Console(width=400, color_system="standard")
SCHED = ("#e0f2f1", "#00695c") # scheduler (teal)
COORD = ("#ede7f6", "#5e35b1") # coordinator layer (purple)
SUP = ("#e3f2fd", "#1565c0") # supervisor (blue)
-JVM = ("#fbe9e7", "#d84315") # JVM runtime + user code (orange)
+LANG = ("#fbe9e7", "#d84315") # language runtime + user code — JVM or
compiled binary (orange)
API = ("#fdecea", "#c62828") # execution API (red)
-SCHED_C, COORD_C, SUP_C, JVM_C, API_C = SCHED[1], COORD[1], SUP[1], JVM[1],
API[1]
+SCHED_C, COORD_C, SUP_C, LANG_C, API_C = SCHED[1], COORD[1], SUP[1], LANG[1],
API[1]
LIFELINE = "#b0bec5"
@@ -73,8 +77,8 @@ LIFELINE = "#b0bec5"
# drawn between neighboring lifelines.
PARTICIPANTS = [
("sched", "Scheduler", "creates the ExecuteTask workload", SCHED),
- ("jvm", "JVM subprocess", "Server · Task · user code", JVM),
- ("sup", "Supervisor (Python)", "Coordinator · _JavaActivitySubprocess",
SUP),
+ ("lang", "Language subprocess", "bundle entry point · task · user code",
LANG),
+ ("sup", "Supervisor (Python)", "Coordinator · _PopenActivitySubprocess",
SUP),
("api", "Execution API", "FastAPI · TEI / AIP-72", API),
]
@@ -86,26 +90,27 @@ STEPS: list[dict[str, Any]] = [
"from": "sched",
"to": "sup",
"color": SCHED_C,
- "label": 'ExecuteTask workload\n(carries queue="java")',
+ "label": "ExecuteTask workload\n(carries queue java / golang)",
},
{
"kind": "self",
"actor": "sup",
"theme": COORD,
- "text": 'CoordinatorManager.for_queue("java") → JavaCoordinator\nopen
two loopback-TCP servers',
+ "text": "CoordinatorManager.for_queue(queue) → SubprocessCoordinator\n"
+ "(JavaCoordinator / ExecutableCoordinator) · open two loopback-TCP
servers",
},
{
"kind": "msg",
"from": "sup",
- "to": "jvm",
+ "to": "lang",
"color": COORD_C,
- "label": "subprocess.Popen(java -jar bundle)\n(spawn the JVM)",
+ "label": "spawn bundle subprocess (--comm / --logs)\nJava: java
-classpath … · Go: exec packed binary",
},
{
"kind": "msg",
- "from": "jvm",
+ "from": "lang",
"to": "sup",
- "color": JVM_C,
+ "color": LANG_C,
"label": "TCP connect back\n(comm + logs channels)",
},
{"kind": "msg", "from": "sup", "to": "api", "color": SUP_C, "label":
"PATCH .../run\n(TI started)"},
@@ -120,16 +125,21 @@ STEPS: list[dict[str, Any]] = [
{
"kind": "msg",
"from": "sup",
- "to": "jvm",
+ "to": "lang",
"color": SUP_C,
"label": "StartupDetails (msgpack over TCP)\n→ build Context",
},
- {"kind": "self", "actor": "jvm", "theme": JVM, "text":
"Task.execute(Context, Client) [USER CODE]"},
+ {
+ "kind": "self",
+ "actor": "lang",
+ "theme": LANG,
+ "text": "run the task function [USER CODE]\n(Java Task.execute / Go
task func)",
+ },
{
"kind": "msg",
- "from": "jvm",
+ "from": "lang",
"to": "sup",
- "color": JVM_C,
+ "color": LANG_C,
"label": "getConnection / getVariable / getXCom /
setXCom\n(_RequestFrame, msgpack over TCP)",
},
{
@@ -143,15 +153,15 @@ STEPS: list[dict[str, Any]] = [
{
"kind": "msg",
"from": "sup",
- "to": "jvm",
+ "to": "lang",
"color": SUP_C,
"label": "*Result (msgpack over TCP)\nabove 4 messages repeat per
lookup",
},
{
"kind": "msg",
- "from": "jvm",
+ "from": "lang",
"to": "sup",
- "color": JVM_C,
+ "color": LANG_C,
"label": "final state (success / failed / up-for-retry)\nmsgpack over
TCP",
},
{"kind": "msg", "from": "sup", "to": "api", "color": SUP_C, "label":
"PATCH .../state\n(+ upload logs)"},
@@ -161,7 +171,7 @@ STEPS: list[dict[str, Any]] = [
"to": "sched",
"color": SCHED_C,
"style": "dashed",
- "label": "JVM process exits →\nexecute_task() returns the exit code",
+ "label": "language subprocess exits →\nexecute_task() returns the exit
code",
},
]
@@ -265,9 +275,9 @@ def _build_sequence(g) -> None:
def _legend(g) -> None:
entries = [
("lg_sched", "Scheduler → Supervisor", SCHED),
- ("lg_coord", "Coordinator layer (CoordinatorManager /
JavaCoordinator)", COORD),
- ("lg_sup", "Supervisor → JVM / Execution API", SUP),
- ("lg_jvm", "JVM (user code) → Supervisor (loopback TCP)", JVM),
+ ("lg_coord", "Coordinator layer (CoordinatorManager /
SubprocessCoordinator)", COORD),
+ ("lg_sup", "Supervisor → subprocess / Execution API", SUP),
+ ("lg_lang", "Language subprocess (user code) → Supervisor (loopback
TCP)", LANG),
("lg_api", "Execution API → Supervisor (response)", API),
]
with g.subgraph(name="cluster_legend") as legend:
@@ -299,11 +309,11 @@ def _legend(g) -> None:
g.edge(f"sched__{len(STEPS) - 1}", "lg_sched", style="invis")
-def generate_java_sdk_execution_sequence_diagram():
+def generate_coordinator_execution_sequence_diagram():
image_file = MY_DIR / f"{MY_FILENAME}.png"
console.print(f"[bright_blue]Generating sequence image {image_file}")
- g = graphviz.Digraph("java_sdk_execution_sequence")
+ g = graphviz.Digraph("coordinator_execution_sequence")
g.attr(
rankdir="TB",
splines="line",
@@ -324,4 +334,4 @@ def generate_java_sdk_execution_sequence_diagram():
if __name__ == "__main__":
- generate_java_sdk_execution_sequence_diagram()
+ generate_coordinator_execution_sequence_diagram()
diff --git
a/contributing-docs/images/diagram_go_edge_worker_architecture.md5sum
b/contributing-docs/images/diagram_go_edge_worker_architecture.md5sum
new file mode 100644
index 00000000000..8ea4f885204
--- /dev/null
+++ b/contributing-docs/images/diagram_go_edge_worker_architecture.md5sum
@@ -0,0 +1 @@
+6f508dad8e802534ed02155b810c7ad8
diff --git a/airflow-core/docs/img/diagram_native_language_sdk_architecture.png
b/contributing-docs/images/diagram_go_edge_worker_architecture.png
similarity index 100%
rename from airflow-core/docs/img/diagram_native_language_sdk_architecture.png
rename to contributing-docs/images/diagram_go_edge_worker_architecture.png
diff --git a/airflow-core/docs/img/diagram_native_language_sdk_architecture.py
b/contributing-docs/images/diagram_go_edge_worker_architecture.py
similarity index 93%
rename from airflow-core/docs/img/diagram_native_language_sdk_architecture.py
rename to contributing-docs/images/diagram_go_edge_worker_architecture.py
index 9f802620ed3..0d8c5ed5e84 100644
--- a/airflow-core/docs/img/diagram_native_language_sdk_architecture.py
+++ b/contributing-docs/images/diagram_go_edge_worker_architecture.py
@@ -22,11 +22,11 @@
# ]
# ///
"""
-Architecture diagram for a native-language (compiled) Task SDK.
+Architecture diagram for the Go SDK's standalone **edge-worker** deployment
mode.
-The Go SDK (``go-sdk/``) is the first implementation; the same shape applies to
-any future compiled-language SDK (e.g. Java). Unlike the Python Task SDK there
is
-**no Python Supervisor and no msgpack stdin socket**:
+This is the Go SDK's alternative to the Coordinator path: instead of a Python
+Supervisor driving the bundle (the approach Java and the Go Coordinator mode
both
+take), there is **no Python Supervisor and no msgpack socket**:
* a long-running compiled **edge worker** (``airflow-go-edge-worker``) *pulls*
work from the **Edge Executor API** over HTTP;
@@ -36,6 +36,9 @@ any future compiled-language SDK (e.g. Java). Unlike the
Python Task SDK there i
the **Execution API** directly over HTTPS — so, unlike the Python task, it
holds
the task JWT itself.
+This mode is missing features the Supervisor provides (see the Go SDK "known
+limitations"), which is why the Coordinator path is recommended.
+
Rendered with graphviz directly so every label sits inside a sized shape:
3-D box = native OS process, rounded box = an object inside a process,
component = a server app, cylinder = the database, note = a caption.
@@ -85,11 +88,11 @@ def _node(g, node_id: str, title: str, sub: str, *, shape:
str, theme: tuple[str
)
-def generate_native_language_sdk_architecture_diagram():
+def generate_go_edge_worker_architecture_diagram():
image_file = MY_DIR / f"{MY_FILENAME}.png"
console.print(f"[bright_blue]Generating architecture image {image_file}")
- g = graphviz.Digraph("native_language_sdk_architecture")
+ g = graphviz.Digraph("go_edge_worker_architecture")
g.attr(
rankdir="TB",
splines="spline",
@@ -255,4 +258,4 @@ def generate_native_language_sdk_architecture_diagram():
if __name__ == "__main__":
- generate_native_language_sdk_architecture_diagram()
+ generate_go_edge_worker_architecture_diagram()
diff --git
a/airflow-core/docs/img/diagram_task_sdk_execution_architecture.md5sum
b/contributing-docs/images/diagram_task_sdk_execution_architecture.md5sum
similarity index 100%
rename from airflow-core/docs/img/diagram_task_sdk_execution_architecture.md5sum
rename to
contributing-docs/images/diagram_task_sdk_execution_architecture.md5sum
diff --git a/airflow-core/docs/img/diagram_task_sdk_execution_architecture.png
b/contributing-docs/images/diagram_task_sdk_execution_architecture.png
similarity index 100%
rename from airflow-core/docs/img/diagram_task_sdk_execution_architecture.png
rename to contributing-docs/images/diagram_task_sdk_execution_architecture.png
diff --git a/airflow-core/docs/img/diagram_task_sdk_execution_architecture.py
b/contributing-docs/images/diagram_task_sdk_execution_architecture.py
similarity index 100%
rename from airflow-core/docs/img/diagram_task_sdk_execution_architecture.py
rename to contributing-docs/images/diagram_task_sdk_execution_architecture.py
diff --git a/airflow-core/docs/img/diagram_task_sdk_execution_sequence.md5sum
b/contributing-docs/images/diagram_task_sdk_execution_sequence.md5sum
similarity index 100%
rename from airflow-core/docs/img/diagram_task_sdk_execution_sequence.md5sum
rename to contributing-docs/images/diagram_task_sdk_execution_sequence.md5sum
diff --git a/airflow-core/docs/img/diagram_task_sdk_execution_sequence.png
b/contributing-docs/images/diagram_task_sdk_execution_sequence.png
similarity index 100%
rename from airflow-core/docs/img/diagram_task_sdk_execution_sequence.png
rename to contributing-docs/images/diagram_task_sdk_execution_sequence.png
diff --git a/airflow-core/docs/img/diagram_task_sdk_execution_sequence.py
b/contributing-docs/images/diagram_task_sdk_execution_sequence.py
similarity index 100%
rename from airflow-core/docs/img/diagram_task_sdk_execution_sequence.py
rename to contributing-docs/images/diagram_task_sdk_execution_sequence.py
