Hello Everyone,
TL;DR; I was just about to start to work on a small set of Hooks -
dedicated to retrieving screts from the Secret Backend. I discussed it with Ash
and Kamil
<https://apache-airflow.slack.com/archives/C0145R4NPS5/p1589805908013700> on
Slack today. So far I thought I treat them as usual providers, but Ash
raised some valid concenrs. so I wanted to raise teh proposal before I
start working on it/
*Context:*
Currently we have "Secret Backend" support built in in 2.0 and 1.10.10+. It
includes retrieving the variable and connections (via Secret Manager class)
for:
- Hashicorp Vault
- Secret Manager
- KMS
- AWS secret manager
Those secret managers are configured in:
[secret]
backend=<SecretManagerClass>
backend_kwargs={}
Those are available for use in a nice way (via Jinja templates and the
like), but they need support in the Core of Airlfow (so require 1.10.10+).
This means that if you are on pre 1.10.10 you cannot use those secrets.
Currently you can only use one secret per whole Airflow installation so if
youre secrets are split between several secret managers (or if secrets for
particular service require different credentials) - you cannot use the
mechanism to access such distributed secrets. It's not often case, but I
very well imagine it might happen that there are different sets of
credentials to access different secrets - some services might have
different scopes/level of access needed. .
*Proposal*
We have an idea that we might want also (on top of the above SecretManager
implementation) define generic Hooks for accessing secrets from those
services (just generic secrets, not connection, variables). Simply treat
each of the backends above as another "provider" and create a Hook to
access the service. Such Hook could have just one method:
def get_secret(self, path_prefix: str, secret_id: str) -> Optional[str]
It would use a connection defined (as usual) in ENV variables or database
of Airflow to authenticate with the secret service and retrieve the
secrets.
The good thing with it is that you could have easily multiple secret
backends configured to retrieve secrets for specific "service" (so that you
could keep "generic airflow's secerts" in one backend but still have
possibility of custom operators to use other backends (with different
authentication, scopes etc.). And it is not touching any of the "core" of
Airflow. It's just a set of hooks with corresponding connections that work
the same way as accessing any other provider in Airflow. No core of Airflow
will be touched with this change.
*Pros/Cons*
*Con:*
I do realise it is a bit of duplication in functionality. We already have a
way to connect to a secret backend via airflow configuration and we should
likely promote it rather than introduce additional mechanism.
*Pros:*
* Most of all -> it adds flexibility of accessing several secret backends
for different use-cases. I looked at it so far in the way those hooks are
merely another set of "provider hooks". For me this is nothing different
than "providers" for any other services we have. fFr example "cloudant"
provider has only "CloudantHook" that other custom operators can use. And I
well imagine this might be actually even more convenient to configure
connections in the DB and access secrets this way rather than having to
configure Secret Backends in Airflow configuration.
* The dupication there it is very, very limited (basically a method call to
secret backend).
* Another benefit of it is that it would allow people still stuck on pre
1.10.10 to write custom operators that would like to use secret backends
(via backport operators). And still continue doing it in the future
(possibly migrating to 2.0/1.10.10+ in cases when there is one secret
backed only - but continue ot use connections/hooks where some specific
secrets shoudl be kept in different secret backend.
I would like to hear your opinion on that.
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
