jason810496 commented on code in PR #67153: URL: https://github.com/apache/airflow/pull/67153#discussion_r3265827602
########## go-sdk/adr/0004-self-contained-executable-bundle.md: ########## @@ -0,0 +1,304 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 4. Self-contained executable bundle (footer-embedded source and metadata) + +Date: 2026-05-04 + +## Status + +Accepted. Supersedes the ZIP-archive container portion of +[ADR 0001](0001-bundle-packing-options.md) and the ZIP output sketched +in [ADR 0002](0002-use-go-tool-directive-for-bundle-packer.md). The +packer mechanism (Option A standalone packer + Option D introspection +contract + Option H `tool` directive) is unchanged; only the artefact +the packer writes is changed. + +## Context + +ADR 0001 / ADR 0002 picked a ZIP archive as the bundle container, +following the executable provider's existing +[`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst). +A conforming bundle today is `bundle.zip` with three required entries: +`airflow-metadata.yaml`, the primary DAG source file, and the compiled +executable. + +That layout has three properties we want to preserve: + +1. **Discovery without execution.** The scanner must be able to read + `dag_id` / `task_id` and the SDK language/version from a bundle on + disk without running the binary. ADR 0002 already enforces this — + `airflow-go-pack` runs the binary once at build time, captures its + `--dump-bundle-spec` output into the manifest, and the scanner reads + the manifest at deploy time. +2. **Source available for the UI.** The Airflow UI's source-view + panel needs to render the DAG file. The current spec ships it as a + verbatim ZIP entry referenced by the manifest's `source` field. +3. **Single deployment unit.** Drop one file in + `[executable] bundles_folder` and the scanner picks it up. + +What the ZIP container costs us: + +- **Two artefacts in flight.** `go build` produces a binary; the + packer wraps it into a ZIP. Anything that touches the binary after + it is wrapped (re-strip, re-sign, swap-in a debug build) drifts from + the manifest unless the wrapping is redone. The wrapping step is + cheap but the drift mode is real. +- **A second container format on the consumer side.** The scanner + must open archives, find members by name, and materialise the + executable into a transient cache before the runtime can exec it. + That is `archive/zip` on the Python side plus a per-bundle cache + directory. +- **Inspection requires a different tool than running.** `unzip` to + inspect, then run; or run, then `unzip` to debug. Two muscle memories. + +Native-executable SDKs (Go, Rust, C++, Zig) all produce a single +self-contained binary by static linking. The binary itself is already +the only thing that has to land on the worker host to run a task. The +manifest and the source file are small data the scanner needs but the +runtime doesn't. Both can ride along in a footer appended to the +binary, with the binary remaining a runnable executable. + +This is the same pattern self-extracting installers, `goreleaser`-style +self-update images, and embedded-asset binaries already use: append +data after the OS-recognised binary structure, leave a fixed-size +trailer at the very end so a reader can locate the data, and validate +with a magic value. + +The user-facing claim becomes "the executable *is* the bundle." A +bundle directory looks like: + +``` +/opt/airflow/executable-bundles/ +├── example +├── pipeline +└── analytics +``` + +(Filenames follow OS conventions: no extension on Linux/macOS, `.exe` +on Windows. The scanner identifies bundles by the trailer's magic, not +by the filename.) + +## Decision + +Replace the bundle's ZIP container with a footer appended to the +compiled executable. The executable's normal byte content is unchanged +and it remains directly runnable; the footer is data that follows the +last byte the OS loader cares about. + +### Footer layout + +A bundle file is laid out as: + Review Comment: The user-facing doc regarding the executable bundle spec was added in https://github.com/apache/airflow/pull/67161/changes/478edabcd14b71fdbd4596ea4521ca26047a7a5b#diff-0f63500bac12820edd728da72331356b8546a8d571ed949df07a96e1085b835d -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
