Hi Jarek, I would like if we keep "Secrets" as a separate concept and not mix it with "hooks".
While introducing and reviewing the initial PRs and AIP about it, it made sense to have "Secrets" as a separate concept altogether. "Secrets" is what the Hook would use to interact with external services. As soon as we try to mix those concepts it starts getting unnecessarily complicated. Example: Hashicorp Vault is what I use to get my Airflow Connections that is used in my hooks and operators. But if I need to define Connection for the Secrets that I used to get Connections, it gets "tricky". I would treat getting "Secrets" as the same way as we connect to our "SQL Backend" (sql_alchemy_conn in airflow.cfg). One of the main benefits of the Secrets Backend was we do not need to define anything Connections at all in the Metadata DB. And automation tools can handle airflow.cfg as a single point for this configs. We should not also try to backport Core Services to older versions. I feel very strongly about it and feel "Secrets" concepts is a part of a core functionality rather than a hook. For users > 1.10.10, it is already very easy to write their own secrets backend. We should I think try to get Airflow 2.0 soon'ish with all the features we want to. Note: I am not at my 100% today and if I have misunderstood something I will respond back in coming days. Regards, Kaxil On Mon, May 18, 2020, 14:46 Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Hello Everyone, > > TL;DR; I was just about to start to work on a small set of Hooks - > dedicated to retrieving screts from the Secret Backend. I discussed it > with Ash > and Kamil > <https://apache-airflow.slack.com/archives/C0145R4NPS5/p1589805908013700> > on > Slack today. So far I thought I treat them as usual providers, but Ash > raised some valid concenrs. so I wanted to raise teh proposal before I > start working on it/ > > *Context:* > > Currently we have "Secret Backend" support built in in 2.0 and 1.10.10+. It > includes retrieving the variable and connections (via Secret Manager class) > for: > > - Hashicorp Vault > - Secret Manager > - KMS > - AWS secret manager > > Those secret managers are configured in: > > [secret] > backend=<SecretManagerClass> > backend_kwargs={} > > Those are available for use in a nice way (via Jinja templates and the > like), but they need support in the Core of Airlfow (so require 1.10.10+). > This means that if you are on pre 1.10.10 you cannot use those secrets. > Currently you can only use one secret per whole Airflow installation so if > youre secrets are split between several secret managers (or if secrets for > particular service require different credentials) - you cannot use the > mechanism to access such distributed secrets. It's not often case, but I > very well imagine it might happen that there are different sets of > credentials to access different secrets - some services might have > different scopes/level of access needed. . > > *Proposal* > > We have an idea that we might want also (on top of the above SecretManager > implementation) define generic Hooks for accessing secrets from those > services (just generic secrets, not connection, variables). Simply treat > each of the backends above as another "provider" and create a Hook to > access the service. Such Hook could have just one method: > > def get_secret(self, path_prefix: str, secret_id: str) -> Optional[str] > > It would use a connection defined (as usual) in ENV variables or database > of Airflow to authenticate with the secret service and retrieve the > secrets. > > The good thing with it is that you could have easily multiple secret > backends configured to retrieve secrets for specific "service" (so that you > could keep "generic airflow's secerts" in one backend but still have > possibility of custom operators to use other backends (with different > authentication, scopes etc.). And it is not touching any of the "core" of > Airflow. It's just a set of hooks with corresponding connections that work > the same way as accessing any other provider in Airflow. No core of Airflow > will be touched with this change. > > *Pros/Cons* > > *Con:* > > I do realise it is a bit of duplication in functionality. We already have a > way to connect to a secret backend via airflow configuration and we should > likely promote it rather than introduce additional mechanism. > > *Pros:* > > * Most of all -> it adds flexibility of accessing several secret backends > for different use-cases. I looked at it so far in the way those hooks are > merely another set of "provider hooks". For me this is nothing different > than "providers" for any other services we have. fFr example "cloudant" > provider has only "CloudantHook" that other custom operators can use. And I > well imagine this might be actually even more convenient to configure > connections in the DB and access secrets this way rather than having to > configure Secret Backends in Airflow configuration. > > * The dupication there it is very, very limited (basically a method call to > secret backend). > > * Another benefit of it is that it would allow people still stuck on pre > 1.10.10 to write custom operators that would like to use secret backends > (via backport operators). And still continue doing it in the future > (possibly migrating to 2.0/1.10.10+ in cases when there is one secret > backed only - but continue ot use connections/hooks where some specific > secrets shoudl be kept in different secret backend. > > I would like to hear your opinion on that. > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
