Re: [PROPOSAL] Secret Backend Hooks

Cool. I thought it's a misunderstanding :). Great it is clear now! J. On Tue, May 19, 2020 at 11:17 AM Ash Berlin-Taylor wrote: > Yes, sorry I got completely the wrong idea somehow. This makes sense, > and thank you for patiently explaining it to me until I got it! > > My main reason for quest

Re: [PROPOSAL] Secret Backend Hooks

Yes, sorry I got completely the wrong idea somehow. This makes sense, and thank you for patiently explaining it to me until I got it! My main reason for questioning was not this specific feature, but the gradual "scope creep" of Airflow operators. One of the hardest things we as project "stewards

Re: [PROPOSAL] Secret Backend Hooks

Yep. That is exactly why I think we need the hooks. Is it possible you donate your code for the Vault Hook implementation ? I would love to use it for my implementation. (And make you or whoever the author is as co-author :) J. wt., 19 maj 2020, 09:41 użytkownik Nathan Hadfield napisał: > Jar

Re: [PROPOSAL] Secret Backend Hooks

Jarek, We are already using the secret backend for Airflow variables. But, because of the example I explained and also a programmatic need to update our GCP Airflow connections every day, then we still have to maintain a secondary, custom method for Vault authentication and manipulation of oth

Re: [PROPOSAL] Secret Backend Hooks

Let me start again from scratch and use KMS as an example. Maybe - again - we understand things differently: Just to start KMSHook -> has two methods: "encrypt" and "decrypt". I would continue to use that as a base. Again let me repeat that. I do not want to implement a generic SecretHook! I al

Re: [PROPOSAL] Secret Backend Hooks

This is why I was asking for a concrete example of where you'd want to use this, right now I still can't see what problem you are aiming to solve with this proposal. So I'll ask again Jarek: Do you have a concrete use case that is driving you wanting to create SecretManagerHook? > Straightforwar

Re: [PROPOSAL] Secret Backend Hooks

On Mon, May 18, 2020 at 11:17 PM Ash Berlin-Taylor wrote: > > GCP Secret Manager Hook, Vault Hook, KMS Hook, AWS Secret Hook > > Why do we even need Hooks for those? Why can't we use the existing GCP > Secret Manager class inside a custom operator? What does creating a hook > give us? > The same

Re: [PROPOSAL] Secret Backend Hooks

> GCP Secret Manager Hook, Vault Hook, KMS Hook, AWS Secret Hook Why do we even need Hooks for those? Why can't we use the existing GCP Secret Manager class inside a custom operator? What does creating a hook give us? -a On May 18 2020, at 9:50 pm, Jarek Potiuk wrote: >> Are we all talking abo

Re: [PROPOSAL] Secret Backend Hooks

> Are we all talking about different things ? Good point. I think that's the main source of confusion here and we think about different things. > So what I feel that the use case that Nathan defined can just be solved a > VaultHook & VaultOperator for example. That's what I was talking (from the

Re: [PROPOSAL] Secret Backend Hooks

Are we all talking about different things 😁 ? So what I feel that the use case that Nathan defined can just be solved a VaultHook & VaultOperator for example. This should not be confused with "Secrets" at all. Why do we need to create a generic Hooks for all Secrets Backend? Consider we use Post

Re: [PROPOSAL] Secret Backend Hooks

Thanks Nathan, I think your case is really good example where the Hook might be really useful (and apparently somebody did it already via Hooks). I wonder Nathan if you (in the future) switch to secret backend - would you use the same secret backend for Airlfow connections/variables? Or do you fo

Re: [PROPOSAL] Secret Backend Hooks

Yep, I understand. I wasn't necessarily advocating for a Vault hook; just wanted to give some real world colour to the conversation and what we did to solve our needs prior to the secrets backend. I'm sure that extending the class would also enable the same functionality. Cheers, Nathan On

Re: [PROPOSAL] Secret Backend Hooks

Accessing things that aren't connections or variables is, essentially creating a third class of thing that Secrets store. But that is a separate issue to what Jarek is proposing, which is Hooks. For your use case a Python operator sounds like the best fit. A hook is going to have to target the lo

Re: [PROPOSAL] Secret Backend Hooks

Hey, My quick two cents are that it would be good to access secrets that are not explicitly either connections or variables We have a need for DAGs that feature more complex interactions with Vault - which typically end up being custom operators - that I think would be helped by more gener

Re: [PROPOSAL] Secret Backend Hooks

> The good thing with it is that you could have easily multiple secret > backends configured to retrieve secrets for specific "service" (so > that you > could keep "generic airflow's secerts" in one backend but still have > possibility of custom operators to use other backends (with different > a

Re: [PROPOSAL] Secret Backend Hooks

Hi Jarek, I would like if we keep "Secrets" as a separate concept and not mix it with "hooks". While introducing and reviewing the initial PRs and AIP about it, it made sense to have "Secrets" as a separate concept altogether. "Secrets" is what the Hook would use to interact with external servic

[PROPOSAL] Secret Backend Hooks

Hello Everyone, TL;DR; I was just about to start to work on a small set of Hooks - dedicated to retrieving screts from the Secret Backend. I discussed it with Ash and Kamil on Slack today. So far I thought I treat them as us