uranusjr commented on code in PR #74698: URL: https://github.com/apache/airflow/pull/74698#discussion_r4344385670
########## airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst: ########## @@ -20,270 +20,464 @@ Go SDK ====== -|experimental| +The Go SDK lets you write Airflow Dags and tasks in Go. There are two ways to use it: -The Go SDK lets you implement Airflow task logic in Go, with native access to the Airflow "model" -(Variables, Connections, and XCom). The Dag and its scheduling remain in Python; individual tasks delegate -to a compiled Go *bundle* that is launched by -:class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator` for each task instance. +* **Dag definition.** The schedule, the tasks, their options and the dependencies between them are all written + in Go, with no Python file involved. See :ref:`go-sdk/dag-definition`. +* **Python Dag with stub TaskHandler.** A Python Dag declares the tasks with ``@task.stub`` and wires them + together, and a Go ``TaskHandler`` implements each one. See :ref:`go-sdk/stub-tasks`. -Because Go is a compiled language, every task must be compiled ahead of time and registered inside a single, -self-contained native executable called a **bundle**. The bundle also embeds the source file of each Dag built with ``airflow.Dag`` -plus the entrypoint file (the one with ``func main``), and a metadata -manifest (the ``dag_id`` and ``task_id`` map) in a footer appended to the executable, so the executable *is* -the bundle: one runnable file to ship, with no separate manifest or archive. The -:ref:`airflow-go-pack <go-sdk/build>` tool builds and packs that bundle. +Both use the same task API and the same build tool, and one bundle can serve both. + +Go is compiled, so your Dags and tasks are built into a single self-contained executable, the **bundle**, with +the :ref:`airflow-go-pack <go-sdk/build>` tool. The bundle is the only file you deploy. + +The SDK is the ``github.com/apache/airflow/go-sdk`` Go module. + +.. seealso:: + + For the full API reference and the released versions, see + `pkg.go.dev <https://pkg.go.dev/github.com/apache/airflow/go-sdk>`__. .. contents:: Contents :local: :depth: 2 -API reference -------------- - -The generated API reference for the Go SDK module, and the list of its released versions, is available on -`pkg.go.dev <https://pkg.go.dev/github.com/apache/airflow/go-sdk>`__. - Prerequisites ------------- -* Go 1.24 or later to build and pack bundles. This is a build-time requirement only; the worker that runs a - packed bundle needs no Go toolchain, because the bundle is a self-contained native executable. -* The packed bundle must be accessible from the Airflow worker and the Dag processor, in the Dag bundle the - coordinator scans, and built for the operating system and CPU architecture of both. -* The ``apache-airflow-task-sdk`` package (installed with Airflow) provides the coordinator; no additional - Python packages are needed. +* Go 1.25 or later to build bundles. Airflow needs no Go toolchain to run them. +* A bundle runs on the Airflow workers and on the Dag processor, so build it for their operating system and CPU + architecture (see :ref:`go-sdk/cross-platform`). +* The ``apache-airflow-task-sdk`` package, installed with Airflow, provides + :class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator`, which runs the bundle. No other Python + package is needed. +* In your Go module, add the SDK, and ``airflow-go-pack`` as a tool: -Execution architecture ----------------------- + .. code-block:: bash -A Python task runner launches the Go bundle directly, with no separate Go worker process on the host. This -is the same coordinator mechanism the Java SDK uses. Because the mature Python supervisor handles the -Airflow-facing concerns, Go tasks inherit remote task logs (S3/GCS), the full range of task states, and -alternate XCom backends rather than implementing them again in Go. + go get github.com/apache/airflow/go-sdk + go get -tool github.com/apache/airflow/go-sdk/cmd/airflow-go-pack .. _go-sdk/quick-start: Quick start ----------- -The following example shows the minimal moving parts: a Python Dag with two stub tasks, and a Go -implementation of those tasks. +This example declares a whole Dag in Go, builds it into a bundle, and deploys it, with no Python file involved. -Python Dag (the scheduling side) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Declare a Dag in ``main.go``: -.. code-block:: python +.. code-block:: go - from airflow.sdk import dag, task + package main + import ( + "log" - @dag - def simple_dag(): - @task.stub(queue="golang") - def extract(): ... + "github.com/apache/airflow/go-sdk/airflow" + ) - @task.stub(queue="golang") - def transform(): ... + func extract(airflow.Context) (int, error) { + return 42, nil + } - extract() >> transform() + func report(actx airflow.Context, rows int) error { + actx.Logger().InfoContext(actx, "extracted rows", "rows", rows) + return nil + } + func main() { + dag := airflow.Dag("go_hello", airflow.DagSpec{Schedule: "@daily", Queue: "golang"}) - simple_dag() + extracted := dag.Task(extract) + dag.Task(report, airflow.Inputs(extracted)) -``@task.stub`` declares the *shape* of the Go tasks (their names and dependencies) without any Python -implementation. The ``queue`` value routes the task to the Go coordinator. + bundle := airflow.Bundle() + bundle.Register(dag) + if err := bundle.Serve(); err != nil { + log.Fatal(err) + } + } -Go implementation -~~~~~~~~~~~~~~~~~~ +Build it into a bundle: -A task is an ordinary Go function whose first parameter is an ``airflow.Context``. Everything Airflow gives -the task is a method on it, so the signature stays the same whatever the task uses. +.. code-block:: bash -.. code-block:: go + go tool airflow-go-pack --output dist/go_hello . - import ( - "runtime" +Put ``dist/go_hello`` in a Dag bundle, for example the default ``dags-folder`` bundle, which reads the +``[core] dags_folder`` directory (see :doc:`Dag bundles </administration-and-deployment/dag-bundles>`). Then send +the ``golang`` queue to the Go coordinator in ``airflow.cfg``: - "github.com/apache/airflow/go-sdk/airflow" - ) +.. code-block:: ini - func extract(actx airflow.Context) (any, error) { - conn, err := actx.Client().GetConnection(actx, "test_http") - if err != nil { - return nil, err - } - actx.Logger().InfoContext(actx, "fetched connection", "host", conn.Host) - // ... do work, honour actx cancellation ... - return map[string]any{"go_version": runtime.Version()}, nil + [sdk] + coordinators = {"go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator"}} + queue_to_coordinator = {"golang": "go"} Review Comment: This exact block appears twice in this document. Can they be unified? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
