Accessing things that aren't connections or variables is, essentially creating a third class of thing that Secrets store.
But that is a separate issue to what Jarek is proposing, which is Hooks. For your use case a Python operator sounds like the best fit. A hook is going to have to target the lowest common denominator, which means vault-specific things are just a needless layer over the top. Extending the existing Secrets Backend interface to support that is doable, but I don't see the need for a Hook. Not everything needs to be a hook :) -ash On May 18 2020, at 4:41 pm, Nathan Hadfield <nathan.hadfi...@king.com> wrote: > Hey, > > > > My quick two cents are that it would be good to access secrets that > are not explicitly either connections or variables > > > > We have a need for DAGs that feature more complex interactions with > Vault - which typically end up being custom operators - that I think > would be helped by more generic capabilities. > > > > For example, we have an automated system that regularly rotates GCP > service accounts across the whole company and stores them in Vault. > We then have to ensure that our different Looker environments always > have these SAs before the old ones expire every 48 hours. To do this, > we wrote a Vault Hook and a Looker Hook and them combine them in an > operator which would read every SA from a specific Vault path and then > update the connection inside Looker. > > > > I don’t know if this will influence your thinking in any way but just > wanted to briefly share our experiences. If anyone would like to > learn more then please reach out and I’d be happy to share more. > > > > Cheers, > > Nathan > > > > On 18/05/2020, 15:21, "Ash Berlin-Taylor" <a...@apache.org> wrote: > > > > > > > The good thing with it is that you could have easily multiple secret > > > backends configured to retrieve secrets for specific "service" (so > > > that you > > > could keep "generic airflow's secerts" in one backend but still have > > > possibility of custom operators to use other backends (with different > > > authentication, scopes etc.). > > > > Having the ability to configure multiple secrets backends is independent > > of this feature. The original PR/AIP to add Secrets Backends > decided to > > leave this ability out as it was more complex to configure. We > could add > > that back in. > > > > I still don't quite get from your example where you are proposing this > > would be used? Can you give a fuller example please? Do you have a > > concrete use case where you need this? > > > > Not everything in Airflow needs to be a hook; just access the secrets > > backend directly. I'm not sure what wrapping an extra layer around these > > classes gives us? > > > > Without a concrete example I can't see anything other than this > adds a > > lot of complexity. > > > > -ash > > > > > > On May 18 2020, at 2:45 pm, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > > > > Hello Everyone, > > > > > > TL;DR; I was just about to start to work on a small set of Hooks - > > > dedicated to retrieving screts from the Secret Backend. I > discussed it > > > with Ash > > > and Kamil > > > > > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__apache-2Dairflow.slack.com_archives_C0145R4NPS5_p1589805908013700&d=DwICaQ&c=-0jfte1J3SKEE6FyZmTngg&r=cgex0jmJ1tJ3A5nVgQ7Pjo7sdo3NkXzIHPolJPlCwBw&m=AIc65Hls2sR87-APqi_0oh4N0F-NOUCC2ulfyS04GGU&s=NBBItsFcPZR-C26VepQEehBPNPEWUsxar_DatX5ulco&e= > > on > > > Slack today. So far I thought I treat them as usual providers, > but Ash > > > raised some valid concenrs. so I wanted to raise teh proposal > before I > > > start working on it/ > > > > > > *Context:* > > > > > > Currently we have "Secret Backend" support built in in 2.0 and > > > 1.10.10+. It > > > includes retrieving the variable and connections (via Secret > Manager class) > > > for: > > > > > > - Hashicorp Vault > > > - Secret Manager > > > - KMS > > > - AWS secret manager > > > > > > Those secret managers are configured in: > > > > > > [secret] > > > backend=<SecretManagerClass> > > > backend_kwargs={} > > > > > > Those are available for use in a nice way (via Jinja templates > and the > > > like), but they need support in the Core of Airlfow (so require > 1.10.10+). > > > This means that if you are on pre 1.10.10 you cannot use those secrets. > > > Currently you can only use one secret per whole Airflow installation > > > so if > > > your secrets are split between several secret managers (or if > secrets for > > > particular service require different credentials) - you cannot > use the > > > mechanism to access such distributed secrets. It's not often > case, but I > > > very well imagine it might happen that there are different sets of > > > credentials to access different secrets - some services might have > > > different scopes/level of access needed. . > > > > > > *Proposal* > > > > > > We have an idea that we might want also (on top of the above > SecretManager > > > implementation) define generic Hooks for accessing secrets from those > > > services (just generic secrets, not connection, variables). > Simply treat > > > each of the backends above as another "provider" and create a > Hook to > > > access the service. Such Hook could have just one method: > > > > > > def get_secret(self, path_prefix: str, secret_id: str) -> Optional[str] > > > > > > It would use a connection defined (as usual) in ENV variables or database > > > of Airflow to authenticate with the secret service and retrieve the > > > secrets. > > > > > > The good thing with it is that you could have easily multiple secret > > > backends configured to retrieve secrets for specific "service" (so > > > that you > > > could keep "generic airflow's secerts" in one backend but still have > > > possibility of custom operators to use other backends (with different > > > authentication, scopes etc.). And it is not touching any of the > > > "core" of > > > Airflow. It's just a set of hooks with corresponding connections > that work > > > the same way as accessing any other provider in Airflow. No core > of Airflow > > > will be touched with this change. > > > > > > *Pros/Cons* > > > > > > *Con:* > > > > > > I do realise it is a bit of duplication in functionality. We already > > > have a > > > way to connect to a secret backend via airflow configuration and > we should > > > likely promote it rather than introduce additional mechanism. > > > > > > *Pros:* > > > > > > * Most of all -> it adds flexibility of accessing several secret backends > > > for different use-cases. I looked at it so far in the way those > hooks are > > > merely another set of "provider hooks". For me this is nothing different > > > than "providers" for any other services we have. fFr example "cloudant" > > > provider has only "CloudantHook" that other custom operators can use. > > > And I > > > well imagine this might be actually even more convenient to configure > > > connections in the DB and access secrets this way rather than > having to > > > configure Secret Backends in Airflow configuration. > > > > > > * The dupication there it is very, very limited (basically a method > > > call to > > > secret backend). > > > > > > * Another benefit of it is that it would allow people still stuck > on pre > > > 1.10.10 to write custom operators that would like to use secret backends > > > (via backport operators). And still continue doing it in the future > > > (possibly migrating to 2.0/1.10.10+ in cases when there is one secret > > > backed only - but continue ot use connections/hooks where some specific > > > secrets shoudl be kept in different secret backend. > > > > > > I would like to hear your opinion on that. > > > > > > J. > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwICaQ&c=-0jfte1J3SKEE6FyZmTngg&r=cgex0jmJ1tJ3A5nVgQ7Pjo7sdo3NkXzIHPolJPlCwBw&m=AIc65Hls2sR87-APqi_0oh4N0F-NOUCC2ulfyS04GGU&s=r-sJroKu0X4XYmnboaHwbpbEIgk5TLwTErkDtwEgvog&e= > > | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwICaQ&c=-0jfte1J3SKEE6FyZmTngg&r=cgex0jmJ1tJ3A5nVgQ7Pjo7sdo3NkXzIHPolJPlCwBw&m=AIc65Hls2sR87-APqi_0oh4N0F-NOUCC2ulfyS04GGU&s=r-sJroKu0X4XYmnboaHwbpbEIgk5TLwTErkDtwEgvog&e= > > > > > >
