neochaotic commented on issue #67599: URL: https://github.com/apache/airflow/issues/67599#issuecomment-5752299635
I hit this exact fork building a Go, Airflow-compatible orchestrator, so here's a data point on "re-implement the masker per language SDK vs. centralize it in the supervisor" — including a case that I think complicates the supervisor-side option. What I landed on was to **not do value-scanning log masking in either place**, and lean on two things instead: 1. **Channel separation.** Keep the data channel (return value / XCom) strictly separate from the log channel, with no automatic data→log crossing. Task return values are never auto-journaled to the log (no `Done. Returned value was: <repr>` line), and the runtime logs kwarg *keys*, not values. That removes the most common leak vectors structurally, with no per-line scanning anywhere. 2. **Name-based masking on read, centrally.** Connection/variable read paths redact by *key name* (`is_sensitive_key → ***`) at serialization time. It's centralized and, importantly, works **without the central component ever needing the secret value**. Two things from that experience that bear on the decision here: - **Supervisor-side value-scanning has a coverage hole for externally-resolved secrets.** When a secret is resolved task/pod-side (e.g. an external secrets backend the control plane deliberately never sees), the supervisor doesn't hold the value and so can't scan-and-redact it. Centralizing in the supervisor only fully works if the supervisor is guaranteed to know every secret value — which stops being true once external/pod-side resolution is in play. - **Structural separation does not cover *derived* secrets.** The one vector neither channel-separation nor name-based masking catches is user code that derives/transforms a secret and then prints it. That specific case is exactly what a value-scanning `SecretsMasker` exists to close — so if it's in scope, *some* component that knows the value has to scan, and per the point above that component often has to be the runtime, not the supervisor. So my takeaway isn't "per-SDK" or "supervisor" — it's that the decision hinges on two prior questions: - **(a)** Are derived-then-printed secrets in scope? If not, you may not need a value-scanning masker at all (channel separation + name-based read masking covers the rest). - **(b)** Is the supervisor guaranteed to know every secret value? If external/pod-side resolution is allowed, no — and then a supervisor-only masker leaves that class uncovered, while the runtime is the only place the value exists. If it's helpful I'm happy to write up the channel-separation design in more detail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
