vincbeck opened a new issue, #65682: URL: https://github.com/apache/airflow/issues/65682
### Description ### Context Multi-team mode ([docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/multi-team.html)) allows organizations to run multiple teams within a single Airflow deployment with resource isolation. When a task requests a Variable or Connection, the secrets backend receives a `team_name` parameter so it can return team-specific values. The built-in secrets backends (environment variables, metastore, local filesystem) already support multi-team. However, the following provider secrets backends accept the `team_name` parameter but **ignore it**, meaning they cannot provide team-scoped secrets: - `AzureKeyVaultBackend` (`providers/microsoft/azure`) - `KubernetesSecretsBackend` (`providers/cncf/kubernetes`) - `LockboxSecretBackend` (`providers/yandex`) ### Current behavior All three backends accept `team_name` in `get_conn_value()` and `get_variable()` but do not use it. For example, in `KubernetesSecretsBackend`: ```python def get_conn_value(self, conn_id: str, team_name: str | None = None) -> str | None: # Multi-team isolation is not currently supported; ``team_name`` is accepted # for API compatibility but ignored. return self._get_secret(self.connections_label, conn_id, self.connections_data_key) ``` ### Expected behavior When `team_name` is provided, each backend should resolve a team-specific secret first, falling back to the global secret if no team-specific one is found. The exact mechanism will differ per backend: #### 1. AzureKeyVaultBackend Use a team-specific prefix to build the secret path. For example, with `connections_prefix="airflow-connections"`, `team_name="team_a"`, and `conn_id="my_db"`: - Look up: `airflow-connections-team-a-my-db` (team-scoped) - Fall back to: `airflow-connections-my-db` (global) The `_is_team_specific_accessed_as_global` guard from `EnvironmentVariablesBackend` should also be applied to prevent team-scoped secrets from being accessed without a team context. #### 2. KubernetesSecretsBackend Use a team-specific label to discover secrets. For example, add a team label (e.g., `airflow.apache.org/team`) and query with both the identifier label and the team label: - Look up: secrets with labels `airflow.apache.org/connection-id=my_db` AND `airflow.apache.org/team=team_a` - Fall back to: secrets with label `airflow.apache.org/connection-id=my_db` and no team label A new constructor parameter (e.g., `team_label`) should be added to configure the team label key. #### 3. LockboxSecretBackend Use a team-specific prefix to build the secret name. For example, with `connections_prefix="airflow/connections"`, `team_name="team_a"`, and `conn_id="my_db"`: - Look up: `airflow/connections/team_a/my_db` (team-scoped) - Fall back to: `airflow/connections/my_db` (global) ### Reference implementation The `EnvironmentVariablesBackend` in `airflow-core/src/airflow/secrets/environment_variables.py` serves as the reference for how multi-team is handled in secrets backends: ```python def get_conn_value(self, conn_id: str, team_name: str | None = None) -> str | None: if self._is_team_specific_accessed_as_global(conn_id, team_name): return None if team_name and ( team_var := os.environ.get(f"{CONN_ENV_PREFIX}_{team_name.upper()}___" + conn_id.upper()) ): return team_var return os.environ.get(CONN_ENV_PREFIX + conn_id.upper()) ``` Key patterns to follow: 1. When `team_name` is provided, try team-specific lookup first, then fall back to global 2. Guard against team-scoped identifiers being accessed without a team context (`_is_team_specific_accessed_as_global`) ### Files to modify - `providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py` - `providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/secrets/kubernetes_secrets_backend.py` - `providers/yandex/src/airflow/providers/yandex/secrets/lockbox.py` - Corresponding test files for each backend ### Acceptance criteria - [ ] `AzureKeyVaultBackend.get_conn_value()` and `get_variable()` resolve team-scoped secrets when `team_name` is provided - [ ] `KubernetesSecretsBackend.get_conn_value()` and `get_variable()` resolve team-scoped secrets when `team_name` is provided - [ ] `LockboxSecretBackend.get_conn_value()` and `get_variable()` resolve team-scoped secrets when `team_name` is provided - [ ] Each backend falls back to global secrets when no team-specific secret is found - [ ] Each backend guards against team-scoped identifiers being accessed without a team context - [ ] Unit tests cover team-specific lookup, global fallback, and the team-specific-accessed-as-global guard for each backend - [ ] Documentation is updated for each backend to describe the team-scoped secret naming convention ### Use case/motivation _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
