jason810496 opened a new pull request, #67315: URL: https://github.com/apache/airflow/pull/67315
# go-sdk: Add coordinator-mode protocol primitives and SDK surface hooks - **Depends on https://github.com/apache/airflow/pull/67153 get merged first** - **Diff for early review**: https://github.com/jason810496/airflow/compare/refactor/go-sdk/adrs...jason810496:refactor/go-sdk/coordinator-protocol - related: split out of https://github.com/apache/airflow/pull/67154 to land it in three reviewable PRs (protocol primitives -> comms layer -> runtime entry point); ADR 0003 (coordinator-protocol msgpack-over-IPC) is defined in https://github.com/apache/airflow/pull/67153. ## Why First PR in a 3-PR stack carved out of #67154 so each piece stays small enough to review on its own. The Python supervisor needs a Go SDK runtime it can launch directly and talk to over a comm socket instead of going through go-plugin / gRPC, but before the runtime itself can land, the wire-level pieces (frame I/O, message envelopes) and a couple of small bundle/sdk surface hooks need to be in place; doing them in a separate PR keeps the runtime PR focused on the dispatch loop rather than on protocol minutiae. This PR is scaffolding only -- no coordinator entry point is wired here, so existing go-plugin / Edge Worker behaviour is unchanged. Dag-file parsing over the same protocol is intentionally out of scope for this stack and will land in a follow-up once that format settles. ## How - `pkg/execution/frames.go` -- length-prefixed msgpack frame reader/writer with a `maxFrameSize` cap so a corrupted length header can't make the runtime allocate unbounded memory. - `pkg/execution/messages.go` -- typed envelopes for every supervisor / runtime message the task-execution path uses (`StartupDetails`, `GetVariable`, `GetConnection`, XCom push/pull, `TaskStateMsg`, ...) so the rest of the runtime code is enum-driven rather than `map[string]any`-driven. - `bundle/bundlev1/task.go` -- checks `sdkcontext.SdkClientContextKey` before constructing the default `sdk.NewClient()`. The injection site lives here (not in the runtime package) because `taskFunction.Execute` is the only place the client is built for a task; the coordinator runtime in the last PR of this stack sets the key, and every other caller silently falls through to the existing default. - `sdk/sdk.go` -- documents `VariableClient` as two single-purpose methods (`GetVariable` for the raw string, `UnmarshalJSONVariable` for JSON-encoded values) so callers stop having to guess which one applies. - `sdk/connection.go` + `sdk/client.go` -- renames `connFromAPIResponse` to `ConnFromAPIResponse` so future internal SDK packages can reuse the existing conversion. Leaves the two `// TODO: register with SecretsMasker` markers on `GetVariable` and `GetConnection`: this PR intentionally doesn't ship secret masking; both halves of the masker integration (logger side + secret-source side) will land together in a later change. - `pkg/sdkcontext/keys.go` -- adds `SdkClientContextKey`, used by the injection site above. - `go.mod` / `go.sum` -- adds `github.com/vmihailenco/msgpack/v5`, the encoding the supervisor speaks. ## What - Add `go-sdk/pkg/execution/{frames,messages}.go` plus matching unit tests. - Extend `go-sdk/bundle/bundlev1/task.go` to honour `sdkcontext.SdkClientContextKey` when building a task's `sdk.Client`. - Tweak `go-sdk/sdk/{client,connection,sdk}.go` (export `ConnFromAPIResponse`, document `VariableClient`, leave masker TODOs). - Add `SdkClientContextKey` in `go-sdk/pkg/sdkcontext/keys.go`. - Bump `go-sdk/{go.mod,go.sum}` for `github.com/vmihailenco/msgpack/v5`. ## Next - refactor/go-sdk/coordinator-comms (PR #TBD, opened after this one merges) --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Opus 4.7 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
