Hi All, I would like to propose a change to how the task SDK loads secrets backends, and get objections (if any).
Getting a connection or a variable today constructs a brand new backend instance. Any authenticated client the backend holds is discarded and rebuilt each time a secret is accessed. For an external backend, that means re-authenticating per secret. With HashiCorp Vault, one AppRole login for every variable a task reads. I measured two variables fetched in one process producing two logins. I propose memoising the backend list where it is constructed, keyed on the search path. Keyed rather than a single global because core's cache only ever serves the server chain. Every other search path takes its rebuild branch, and the task runner always asks for the worker chain, so copying core's shape verbatim would leave worker lookups rebuilding exactly as they do now. This change means that holding an instance for the life of a task process will mean that an expiring credential has to refresh. I went through all seven external backends in our providers: - amazon, google, microsoft/azure: auth is delegated to boto3, google-auth and azure-identity, all of which refresh credentials internally. - akeyless: keeps its own token cache with an explicit TTL and refreshes. - yandex: the Yandex SDK refreshes from a service account key, and raw OAuth tokens are long-lived. - hashicorp: covered by a provider change that just went in, which drops two redundant auth/token/lookup-self calls per lookup and re-authenticates reactively on a 403 instead. (PR: https://github.com/apache/airflow/pull/71701) - cncf/kubernetes: not covered. It calls load_incluster_config() inside a cached_property, so the projected service account token is read once. Bound tokens expire and kubelet rotates the file, and the Python client does not re-read it. The Kubernetes gap already exists on the server side, where core has cached backend instances for the life of the process for years, so this is not new. It does become reachable in a worker whose task outlives the token. If someone is interested in fixing it like how Vault did it, be my guest. *What am I asking?* Any objection to backend instances living for the life of the process on the worker path, given that is already the case on the server path? And does anyone know of a backend, in-tree or out, that would not tolerate it? Out-of-tree backends are the case I cannot audit. Thanks & Regards, Amogh Desai
