Hey,
My quick two cents are that it would be good to access secrets that are not
explicitly either connections or variables
We have a need for DAGs that feature more complex interactions with Vault -
which typically end up being custom operators - that I think would be helped by
more generic capabilities.
For example, we have an automated system that regularly rotates GCP service
accounts across the whole company and stores them in Vault. We then have to
ensure that our different Looker environments always have these SAs before the
old ones expire every 48 hours. To do this, we wrote a Vault Hook and a Looker
Hook and them combine them in an operator which would read every SA from a
specific Vault path and then update the connection inside Looker.
I don’t know if this will influence your thinking in any way but just wanted to
briefly share our experiences. If anyone would like to learn more then please
reach out and I’d be happy to share more.
Cheers,
Nathan
On 18/05/2020, 15:21, "Ash Berlin-Taylor" <a...@apache.org> wrote:
> The good thing with it is that you could have easily multiple secret
> backends configured to retrieve secrets for specific "service" (so
> that you
> could keep "generic airflow's secerts" in one backend but still have
> possibility of custom operators to use other backends (with different
> authentication, scopes etc.).
Having the ability to configure multiple secrets backends is independent
of this feature. The original PR/AIP to add Secrets Backends decided to
leave this ability out as it was more complex to configure. We could add
that back in.
I still don't quite get from your example where you are proposing this
would be used? Can you give a fuller example please? Do you have a
concrete use case where you need this?
Not everything in Airflow needs to be a hook; just access the secrets
backend directly. I'm not sure what wrapping an extra layer around these
classes gives us?
Without a concrete example I can't see anything other than this adds a
lot of complexity.
-ash
On May 18 2020, at 2:45 pm, Jarek Potiuk <jarek.pot...@polidea.com> wrote:
> Hello Everyone,
>
> TL;DR; I was just about to start to work on a small set of Hooks -
> dedicated to retrieving screts from the Secret Backend. I discussed it
> with Ash
> and Kamil
>
<https://urldefense.proofpoint.com/v2/url?u=https-3A__apache-2Dairflow.slack.com_archives_C0145R4NPS5_p1589805908013700&d=DwICaQ&c=-0jfte1J3SKEE6FyZmTngg&r=cgex0jmJ1tJ3A5nVgQ7Pjo7sdo3NkXzIHPolJPlCwBw&m=AIc65Hls2sR87-APqi_0oh4N0F-NOUCC2ulfyS04GGU&s=NBBItsFcPZR-C26VepQEehBPNPEWUsxar_DatX5ulco&e=
> on
> Slack today. So far I thought I treat them as usual providers, but Ash
> raised some valid concenrs. so I wanted to raise teh proposal before I
> start working on it/
>
> *Context:*
>
> Currently we have "Secret Backend" support built in in 2.0 and
> 1.10.10+. It
> includes retrieving the variable and connections (via Secret Manager
class)
> for:
>
> - Hashicorp Vault
> - Secret Manager
> - KMS
> - AWS secret manager
>
> Those secret managers are configured in:
>
> [secret]
> backend=<SecretManagerClass>
> backend_kwargs={}
>
> Those are available for use in a nice way (via Jinja templates and the
> like), but they need support in the Core of Airlfow (so require 1.10.10+).
> This means that if you are on pre 1.10.10 you cannot use those secrets.
> Currently you can only use one secret per whole Airflow installation
> so if
> your secrets are split between several secret managers (or if secrets for
> particular service require different credentials) - you cannot use the
> mechanism to access such distributed secrets. It's not often case, but I
> very well imagine it might happen that there are different sets of
> credentials to access different secrets - some services might have
> different scopes/level of access needed. .
>
> *Proposal*
>
> We have an idea that we might want also (on top of the above SecretManager
> implementation) define generic Hooks for accessing secrets from those
> services (just generic secrets, not connection, variables). Simply treat
> each of the backends above as another "provider" and create a Hook to
> access the service. Such Hook could have just one method:
>
> def get_secret(self, path_prefix: str, secret_id: str) -> Optional[str]
>
> It would use a connection defined (as usual) in ENV variables or database
> of Airflow to authenticate with the secret service and retrieve the
> secrets.
>
> The good thing with it is that you could have easily multiple secret
> backends configured to retrieve secrets for specific "service" (so
> that you
> could keep "generic airflow's secerts" in one backend but still have
> possibility of custom operators to use other backends (with different
> authentication, scopes etc.). And it is not touching any of the
> "core" of
> Airflow. It's just a set of hooks with corresponding connections that work
> the same way as accessing any other provider in Airflow. No core of
Airflow
> will be touched with this change.
>
> *Pros/Cons*
>
> *Con:*
>
> I do realise it is a bit of duplication in functionality. We already
> have a
> way to connect to a secret backend via airflow configuration and we should
> likely promote it rather than introduce additional mechanism.
>
> *Pros:*
>
> * Most of all -> it adds flexibility of accessing several secret backends
> for different use-cases. I looked at it so far in the way those hooks are
> merely another set of "provider hooks". For me this is nothing different
> than "providers" for any other services we have. fFr example "cloudant"
> provider has only "CloudantHook" that other custom operators can use.
> And I
> well imagine this might be actually even more convenient to configure
> connections in the DB and access secrets this way rather than having to
> configure Secret Backends in Airflow configuration.
>
> * The dupication there it is very, very limited (basically a method
> call to
> secret backend).
>
> * Another benefit of it is that it would allow people still stuck on pre
> 1.10.10 to write custom operators that would like to use secret backends
> (via backport operators). And still continue doing it in the future
> (possibly migrating to 2.0/1.10.10+ in cases when there is one secret
> backed only - but continue ot use connections/hooks where some specific
> secrets shoudl be kept in different secret backend.
>
> I would like to hear your opinion on that.
>
> J.
>
> --
>
> Jarek Potiuk
> Polidea
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwICaQ&c=-0jfte1J3SKEE6FyZmTngg&r=cgex0jmJ1tJ3A5nVgQ7Pjo7sdo3NkXzIHPolJPlCwBw&m=AIc65Hls2sR87-APqi_0oh4N0F-NOUCC2ulfyS04GGU&s=r-sJroKu0X4XYmnboaHwbpbEIgk5TLwTErkDtwEgvog&e=
> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwICaQ&c=-0jfte1J3SKEE6FyZmTngg&r=cgex0jmJ1tJ3A5nVgQ7Pjo7sdo3NkXzIHPolJPlCwBw&m=AIc65Hls2sR87-APqi_0oh4N0F-NOUCC2ulfyS04GGU&s=r-sJroKu0X4XYmnboaHwbpbEIgk5TLwTErkDtwEgvog&e=
>
>
