dynamic task scenario

2021-06-09 Thread Daniel Standish
*background* Suppose you have a dag with tasks that run some data science model for *active* AB tests. Suppose there are 2-5 tests running at any given time. And the tests come and go every couple weeks. Right now what I'm doing is I have a task at start of dag that updates an airflow variable

Re: Backing up PostgreSQL DB on K8s

2021-06-09 Thread Daniel Standish
> Thanks Daniel. We would need to add this as an option to the K8s Helm Chart and that is kinda outside our current scope. I may be misunderstanding what you're saying, but there's no need to modify the helm chart. Generally speaking all airflow helm charts support using a metastore that is

Re: Backing up PostgreSQL DB on K8s

2021-06-09 Thread Lewis John McGibbney
Hi Sumit, Thanks for the suggestion. We will experiment with extending the Helm chart to bake in in the PVC and PV objects... I can't see anything like this so far digging through the code so I feel it is lacking. Being able to configure this in values.yaml or CLI during install and update

Re: Backing up PostgreSQL DB on K8s

2021-06-09 Thread Lewis John McGibbney
Thanks Daniel. We would need to add this as an option to the K8s Helm Chart and that is kinda outside our current scope. Thanks for the suggestion. lewismc On 2021/06/09 05:46:00, Daniel Standish wrote: > Perhaps it goes without saying but you might consider using cloud sql > option such as